2023-12-24T17:35:01.585 INFO:root:teuthology version: 0.0.1.dev226+g9680d2a 2023-12-24T17:35:01.586 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-12-24_15:40:05-rbd-wip-yuri4-testing-2023-12-21-0917-quincy-distro-default-smithi/7500245 --name yuriw-2023-12-24_15:40:05-rbd-wip-yuri4-testing-2023-12-21-0917-quincy-distro-default-smithi --description rbd/cli_v1/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/format-1 msgr-failures/few objectstore/bluestore-stupid pool/small-cache-pool supported-random-distro$/{centos_8} workloads/rbd_cli_generic} --verbose -- /home/teuthworker/archive/yuriw-2023-12-24_15:40:05-rbd-wip-yuri4-testing-2023-12-21-0917-quincy-distro-default-smithi/7500245/orig.config.yaml 2023-12-24T17:35:01.619 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-24T17:35:01.718 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-24_15:40:05-rbd-wip-yuri4-testing-2023-12-21-0917-quincy-distro-default-smithi/7500245 branch: wip-yuri4-testing-2023-12-21-0917-quincy description: rbd/cli_v1/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/format-1 msgr-failures/few objectstore/bluestore-stupid pool/small-cache-pool supported-random-distro$/{centos_8} workloads/rbd_cli_generic} email: null first_in_suite: false job_id: '7500245' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2023-12-24_15:40:05-rbd-wip-yuri4-testing-2023-12-21-0917-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri4-testing-2023-12-21-0917-quincy ceph: conf: client: rbd default format: 1 global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(CACHE_POOL_NEAR_FULL\) - \(CACHE_POOL_NO_HIT_SET\) sha1: 5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2023-12-21-0917-quincy sha1: 5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 1236 sha1: 5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f sleep_before_teardown: 0 subset: 111/120000 suite: rbd suite_branch: wip-yuri4-testing-2023-12-21-0917-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f targets: smithi087.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC3zm7LNATb0NkKyv/ek111PQVxF5d3AKbhUg0JJ5rq0jQEgeD6U4YLaQhyKi/4V9SdDJvED3VWEN53UE54teL0= tasks: - install: null - ceph: null - exec: client.0: - sudo ceph osd pool create cache 4 - sudo ceph osd tier add rbd cache - sudo ceph osd tier cache-mode cache writeback - sudo ceph osd tier set-overlay rbd cache - sudo ceph osd pool set cache hit_set_type bloom - sudo ceph osd pool set cache hit_set_count 8 - sudo ceph osd pool set cache hit_set_period 60 - sudo ceph osd pool set cache target_max_objects 250 - workunit: clients: client.0: - rbd/cli_generic.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9680d2aba1e76f64ade295c81599c97907457848 timestamp: 2023-12-24_15:40:05 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078 2023-12-24T17:35:01.718 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f/qa; will attempt to use it 2023-12-24T17:35:01.719 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f/qa/tasks 2023-12-24T17:35:01.719 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-24T17:35:01.720 INFO:teuthology.task.internal:Checking packages... 2023-12-24T17:35:01.744 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f' 2023-12-24T17:35:01.744 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-24T17:35:01.744 INFO:teuthology.packaging:ref: None 2023-12-24T17:35:01.744 INFO:teuthology.packaging:tag: None 2023-12-24T17:35:01.744 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-12-21-0917-quincy 2023-12-24T17:35:01.744 INFO:teuthology.packaging:sha1: 5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f 2023-12-24T17:35:01.745 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2023-12-21-0917-quincy 2023-12-24T17:35:01.957 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-168.g5e7ce129 2023-12-24T17:35:01.959 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-24T17:35:01.965 INFO:teuthology.task.internal:no buildpackages task found 2023-12-24T17:35:01.965 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-24T17:35:01.972 INFO:teuthology.task.internal:Saving configuration 2023-12-24T17:35:01.987 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-24T17:35:01.993 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-24T17:35:02.021 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi087.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-24_15:40:05-rbd-wip-yuri4-testing-2023-12-21-0917-quincy-distro-default-smithi/7500245', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-24 17:30:00.411876', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC3zm7LNATb0NkKyv/ek111PQVxF5d3AKbhUg0JJ5rq0jQEgeD6U4YLaQhyKi/4V9SdDJvED3VWEN53UE54teL0='} 2023-12-24T17:35:02.021 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-24T17:35:02.028 INFO:teuthology.task.internal:roles: ubuntu@smithi087.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-12-24T17:35:02.028 INFO:teuthology.run_tasks:Running task console_log... 2023-12-24T17:35:02.066 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff0f5787e50>, signals=[15]) 2023-12-24T17:35:02.066 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-24T17:35:02.072 INFO:teuthology.task.internal:Opening connections... 2023-12-24T17:35:02.072 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi087.front.sepia.ceph.com 2023-12-24T17:35:02.074 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-24T17:35:02.142 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-24T17:35:02.149 DEBUG:teuthology.orchestra.run.smithi087:> uname -m 2023-12-24T17:35:02.214 INFO:teuthology.orchestra.run.smithi087.stdout:x86_64 2023-12-24T17:35:02.215 DEBUG:teuthology.orchestra.run.smithi087:> cat /etc/os-release 2023-12-24T17:35:02.275 INFO:teuthology.orchestra.run.smithi087.stdout:NAME="CentOS Stream" 2023-12-24T17:35:02.276 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION="8" 2023-12-24T17:35:02.276 INFO:teuthology.orchestra.run.smithi087.stdout:ID="centos" 2023-12-24T17:35:02.276 INFO:teuthology.orchestra.run.smithi087.stdout:ID_LIKE="rhel fedora" 2023-12-24T17:35:02.276 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION_ID="8" 2023-12-24T17:35:02.276 INFO:teuthology.orchestra.run.smithi087.stdout:PLATFORM_ID="platform:el8" 2023-12-24T17:35:02.276 INFO:teuthology.orchestra.run.smithi087.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-24T17:35:02.276 INFO:teuthology.orchestra.run.smithi087.stdout:ANSI_COLOR="0;31" 2023-12-24T17:35:02.276 INFO:teuthology.orchestra.run.smithi087.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-24T17:35:02.276 INFO:teuthology.orchestra.run.smithi087.stdout:HOME_URL="https://centos.org/" 2023-12-24T17:35:02.276 INFO:teuthology.orchestra.run.smithi087.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-24T17:35:02.276 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-24T17:35:02.276 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-24T17:35:02.277 INFO:teuthology.lock.ops:Updating smithi087.front.sepia.ceph.com on lock server 2023-12-24T17:35:02.301 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-24T17:35:02.314 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-24T17:35:02.319 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-24T17:35:02.319 DEBUG:teuthology.orchestra.run.smithi087:> test '!' -e /home/ubuntu/cephtest 2023-12-24T17:35:02.339 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-24T17:35:02.345 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-24T17:35:02.345 DEBUG:teuthology.orchestra.run.smithi087:> test -z $(ls -A /var/lib/ceph) 2023-12-24T17:35:02.407 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-24T17:35:02.440 INFO:teuthology.run_tasks:Running task kernel... 2023-12-24T17:35:02.454 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-24T17:35:02.454 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-24T17:35:02.454 DEBUG:teuthology.orchestra.run.smithi087:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-24T17:35:02.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-24T17:35:02.474 DEBUG:teuthology.orchestra.run.smithi087:> uname -r 2023-12-24T17:35:02.534 INFO:teuthology.orchestra.run.smithi087.stdout:4.18.0-489.el8.x86_64 2023-12-24T17:35:02.534 INFO:teuthology.task.kernel:Running kernel on smithi087: 4.18.0-489.el8.x86_64 2023-12-24T17:35:02.534 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum install -y kernel 2023-12-24T17:43:11.518 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - AppStream 73 kB/s | 35 MB 08:06 2023-12-24T17:43:19.743 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - BaseOS 31 MB/s | 57 MB 00:01 2023-12-24T17:43:28.160 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - Extras 49 kB/s | 18 kB 00:00 2023-12-24T17:43:29.077 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - Extras common packages 9.5 kB/s | 6.9 kB 00:00 2023-12-24T17:43:29.378 INFO:teuthology.orchestra.run.smithi087.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2023-12-24T17:43:31.204 INFO:teuthology.orchestra.run.smithi087.stdout:Extra Packages for Enterprise Linux 9.6 MB/s | 16 MB 00:01 2023-12-24T17:43:35.029 INFO:teuthology.orchestra.run.smithi087.stdout:lab-extras 388 kB/s | 24 kB 00:00 2023-12-24T17:43:38.762 INFO:teuthology.orchestra.run.smithi087.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-24T17:43:38.763 INFO:teuthology.orchestra.run.smithi087.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-24T17:43:39.051 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-24T17:43:39.051 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:43:39.052 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2023-12-24T17:43:39.052 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:43:39.052 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-24T17:43:39.052 INFO:teuthology.orchestra.run.smithi087.stdout: kernel x86_64 4.18.0-529.el8 baseos 10 M 2023-12-24T17:43:39.052 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2023-12-24T17:43:39.052 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-core x86_64 4.18.0-529.el8 baseos 43 M 2023-12-24T17:43:39.052 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules x86_64 4.18.0-529.el8 baseos 36 M 2023-12-24T17:43:39.052 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:43:39.053 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-24T17:43:39.053 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:43:39.053 INFO:teuthology.orchestra.run.smithi087.stdout:Install 3 Packages 2023-12-24T17:43:39.053 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:43:39.053 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 89 M 2023-12-24T17:43:39.053 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 96 M 2023-12-24T17:43:39.053 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-24T17:43:39.939 INFO:teuthology.orchestra.run.smithi087.stdout:(1/3): kernel-4.18.0-529.el8.x86_64.rpm 12 MB/s | 10 MB 00:00 2023-12-24T17:43:40.881 INFO:teuthology.orchestra.run.smithi087.stdout:(2/3): kernel-modules-4.18.0-529.el8.x86_64.rpm 20 MB/s | 36 MB 00:01 2023-12-24T17:43:41.399 INFO:teuthology.orchestra.run.smithi087.stdout:(3/3): kernel-core-4.18.0-529.el8.x86_64.rpm 19 MB/s | 43 MB 00:02 2023-12-24T17:43:41.399 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-24T17:43:41.399 INFO:teuthology.orchestra.run.smithi087.stdout:Total 38 MB/s | 89 MB 00:02 2023-12-24T17:43:42.088 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-24T17:43:42.508 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-24T17:43:42.508 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-24T17:43:43.760 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-24T17:43:43.795 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-24T17:43:46.604 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-24T17:43:47.209 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-core-4.18.0-529.el8.x86_64 1/3 2023-12-24T17:43:50.831 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-core-4.18.0-529.el8.x86_64 1/3 2023-12-24T17:43:51.234 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-modules-4.18.0-529.el8.x86_64 2/3 2023-12-24T17:43:54.667 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-modules-4.18.0-529.el8.x86_64 2/3 2023-12-24T17:43:54.964 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-4.18.0-529.el8.x86_64 3/3 2023-12-24T17:44:16.772 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-core-4.18.0-529.el8.x86_64 3/3 2023-12-24T17:44:16.796 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-modules-4.18.0-529.el8.x86_64 3/3 2023-12-24T17:44:17.142 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-4.18.0-529.el8.x86_64 3/3 2023-12-24T17:44:17.142 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-4.18.0-529.el8.x86_64 1/3 2023-12-24T17:44:17.142 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-core-4.18.0-529.el8.x86_64 2/3 2023-12-24T17:44:17.933 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-modules-4.18.0-529.el8.x86_64 3/3 2023-12-24T17:44:17.933 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:44:17.933 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-24T17:44:17.933 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-4.18.0-529.el8.x86_64 kernel-core-4.18.0-529.el8.x86_64 2023-12-24T17:44:17.933 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules-4.18.0-529.el8.x86_64 2023-12-24T17:44:17.933 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:44:17.933 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-24T17:44:18.063 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q kernel | sort -rV | head -n 1 2023-12-24T17:44:18.282 INFO:teuthology.orchestra.run.smithi087.stdout:kernel-4.18.0-529.el8.x86_64 2023-12-24T17:44:18.282 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-529.el8.x86_64 2023-12-24T17:44:18.282 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-529.el8.x86_64 2023-12-24T17:44:18.282 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-24T17:44:18.282 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-24T17:44:18.283 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-12-24T17:44:18.283 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi087.front.sepia.ceph.com, path=None, version=distro) 2023-12-24T17:44:18.283 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q kernel | sort -rV | head -n 1 2023-12-24T17:44:18.490 INFO:teuthology.orchestra.run.smithi087.stdout:kernel-4.18.0-529.el8.x86_64 2023-12-24T17:44:18.491 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-529.el8.x86_64 2023-12-24T17:44:18.491 DEBUG:teuthology.orchestra.run.smithi087:> sudo rpm -qi grub2-tools 2023-12-24T17:44:18.524 INFO:teuthology.orchestra.run.smithi087.stdout:Name : grub2-tools 2023-12-24T17:44:18.524 INFO:teuthology.orchestra.run.smithi087.stdout:Epoch : 1 2023-12-24T17:44:18.524 INFO:teuthology.orchestra.run.smithi087.stdout:Version : 2.02 2023-12-24T17:44:18.524 INFO:teuthology.orchestra.run.smithi087.stdout:Release : 129.el8 2023-12-24T17:44:18.524 INFO:teuthology.orchestra.run.smithi087.stdout:Architecture: x86_64 2023-12-24T17:44:18.525 INFO:teuthology.orchestra.run.smithi087.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-24T17:44:18.525 INFO:teuthology.orchestra.run.smithi087.stdout:Group : System Environment/Base 2023-12-24T17:44:18.525 INFO:teuthology.orchestra.run.smithi087.stdout:Size : 9265586 2023-12-24T17:44:18.525 INFO:teuthology.orchestra.run.smithi087.stdout:License : GPLv3+ 2023-12-24T17:44:18.525 INFO:teuthology.orchestra.run.smithi087.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-24T17:44:18.525 INFO:teuthology.orchestra.run.smithi087.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-24T17:44:18.525 INFO:teuthology.orchestra.run.smithi087.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-24T17:44:18.525 INFO:teuthology.orchestra.run.smithi087.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-24T17:44:18.525 INFO:teuthology.orchestra.run.smithi087.stdout:Relocations : (not relocatable) 2023-12-24T17:44:18.525 INFO:teuthology.orchestra.run.smithi087.stdout:Packager : CentOS BuildSystem 2023-12-24T17:44:18.525 INFO:teuthology.orchestra.run.smithi087.stdout:Vendor : CentOS 2023-12-24T17:44:18.525 INFO:teuthology.orchestra.run.smithi087.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-24T17:44:18.525 INFO:teuthology.orchestra.run.smithi087.stdout:Summary : Support tools for GRUB. 2023-12-24T17:44:18.525 INFO:teuthology.orchestra.run.smithi087.stdout:Description : 2023-12-24T17:44:18.525 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:44:18.525 INFO:teuthology.orchestra.run.smithi087.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-24T17:44:18.526 INFO:teuthology.orchestra.run.smithi087.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-24T17:44:18.526 INFO:teuthology.orchestra.run.smithi087.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-24T17:44:18.526 INFO:teuthology.orchestra.run.smithi087.stdout:hardware devices. 2023-12-24T17:44:18.526 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:44:18.526 INFO:teuthology.orchestra.run.smithi087.stdout:This subpackage provides tools for support of all platforms. 2023-12-24T17:44:18.527 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-24T17:44:18.527 INFO:teuthology.task.kernel:Updating grub on smithi087 to boot 4.18.0-529.el8.x86_64 2023-12-24T17:44:18.527 DEBUG:teuthology.orchestra.run.smithi087:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-24T17:44:20.260 INFO:teuthology.orchestra.run.smithi087.stderr:Generating grub configuration file ... 2023-12-24T17:44:22.853 INFO:teuthology.orchestra.run.smithi087.stderr:done 2023-12-24T17:44:22.858 DEBUG:teuthology.orchestra.run.smithi087:> mktemp 2023-12-24T17:44:22.880 INFO:teuthology.orchestra.run.smithi087.stdout:/tmp/tmp.gKJrCAWpqr 2023-12-24T17:44:22.880 DEBUG:teuthology.orchestra.run.smithi087:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.gKJrCAWpqr 2023-12-24T17:44:22.954 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0666 /tmp/tmp.gKJrCAWpqr 2023-12-24T17:44:23.113 DEBUG:teuthology.orchestra.remote:smithi087:/tmp/tmp.gKJrCAWpqr is 6KB 2023-12-24T17:44:23.160 DEBUG:teuthology.orchestra.run.smithi087:> rm -fr /tmp/tmp.gKJrCAWpqr 2023-12-24T17:44:23.179 DEBUG:teuthology.orchestra.run.smithi087:> sudo /bin/ls /boot/loader/entries || true 2023-12-24T17:44:23.251 INFO:teuthology.orchestra.run.smithi087.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-24T17:44:23.251 INFO:teuthology.orchestra.run.smithi087.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-24T17:44:23.251 INFO:teuthology.orchestra.run.smithi087.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-24T17:44:23.251 INFO:teuthology.orchestra.run.smithi087.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-529.el8.x86_64.conf 2023-12-24T17:44:23.253 DEBUG:teuthology.orchestra.run.smithi087:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-529.el8.x86_64 2023-12-24T17:44:23.409 DEBUG:teuthology.orchestra.run.smithi087:> sudo shutdown -r now 2023-12-24T17:44:53.441 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-24T17:44:53.442 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2023-12-24T17:44:53.443 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-24T17:45:53.502 DEBUG:teuthology.orchestra.remote:timed out 2023-12-24T17:46:02.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2023-12-24T17:46:02.510 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-24T17:46:02.885 DEBUG:teuthology.orchestra.run.smithi087:> true 2023-12-24T17:46:03.170 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi087.front.sepia.ceph.com' 2023-12-24T17:46:03.170 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-529.el8.x86_64"... 2023-12-24T17:46:03.170 DEBUG:teuthology.orchestra.run.smithi087:> uname -r 2023-12-24T17:46:03.187 INFO:teuthology.orchestra.run.smithi087.stdout:4.18.0-529.el8.x86_64 2023-12-24T17:46:03.187 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-529.el8.x86_64 vs 4.18.0-529.el8.x86_64 2023-12-24T17:46:03.187 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-24T17:46:03.187 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-24T17:46:04.189 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-24T17:46:04.189 DEBUG:teuthology.orchestra.run.smithi087:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-24T17:46:04.257 INFO:teuthology.orchestra.run.smithi087.stdout:ttyS1 2023-12-24T17:46:04.276 DEBUG:teuthology.parallel:result is None 2023-12-24T17:46:04.277 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-24T17:46:04.284 INFO:teuthology.task.internal:Creating test directory... 2023-12-24T17:46:04.285 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-24T17:46:04.304 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-24T17:46:04.311 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-24T17:46:04.319 INFO:teuthology.task.internal:Creating archive directory... 2023-12-24T17:46:04.320 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-24T17:46:04.376 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-24T17:46:04.383 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-24T17:46:04.383 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-24T17:46:04.447 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-24T17:46:04.456 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-24T17:46:04.458 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-24T17:46:04.465 INFO:teuthology.task.internal:Configuring sudo... 2023-12-24T17:46:04.465 DEBUG:teuthology.orchestra.run.smithi087:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-24T17:46:04.527 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-24T17:46:04.537 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-24T17:46:04.537 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-24T17:46:04.586 DEBUG:teuthology.orchestra.run.smithi087:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-24T17:46:04.670 DEBUG:teuthology.orchestra.run.smithi087:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-24T17:46:04.720 DEBUG:teuthology.orchestra.run.smithi087:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-24T17:46:04.805 DEBUG:teuthology.orchestra.run.smithi087:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-24T17:46:04.847 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-24T17:46:04.847 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-24T17:46:04.915 DEBUG:teuthology.orchestra.run.smithi087:> sudo service rsyslog restart 2023-12-24T17:46:05.012 INFO:teuthology.orchestra.run.smithi087.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-24T17:46:05.074 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-24T17:46:05.082 INFO:teuthology.task.internal:Starting timer... 2023-12-24T17:46:05.082 INFO:teuthology.run_tasks:Running task pcp... 2023-12-24T17:46:05.153 INFO:teuthology.run_tasks:Running task selinux... 2023-12-24T17:46:05.164 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-12-24T17:46:05.165 DEBUG:teuthology.orchestra.run.smithi087:> sudo service auditd rotate 2023-12-24T17:46:05.263 INFO:teuthology.orchestra.run.smithi087.stdout:Rotating logs: 2023-12-24T17:46:05.264 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-24T17:46:05.265 DEBUG:teuthology.orchestra.run.smithi087:> /usr/sbin/getenforce 2023-12-24T17:46:05.287 INFO:teuthology.orchestra.run.smithi087.stdout:Permissive 2023-12-24T17:46:05.288 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi087.front.sepia.ceph.com': 'permissive'} 2023-12-24T17:46:05.288 DEBUG:teuthology.orchestra.run.smithi087:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-24T17:46:05.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-24T17:46:05.356 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-24T17:46:05.356 DEBUG:teuthology.orchestra.run.smithi087:> sudo /usr/sbin/setenforce permissive 2023-12-24T17:46:05.423 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-24T17:46:05.433 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-12-24T17:46:05.440 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2023-12-24T17:46:05.614 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-24T17:46:05.631 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-24T17:46:05.632 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi087.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-24T17:53:29.190 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi087.front.sepia.ceph.com')] 2023-12-24T17:53:29.192 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2023-12-24T17:53:29.193 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-24T17:53:29.264 DEBUG:teuthology.orchestra.run.smithi087:> true 2023-12-24T17:53:29.332 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi087.front.sepia.ceph.com' 2023-12-24T17:53:29.332 INFO:teuthology.run_tasks:Running task clock... 2023-12-24T17:53:29.342 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-24T17:53:29.343 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-24T17:53:29.343 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-24T17:53:29.404 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-24T17:53:29.419 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-24T17:53:29.494 INFO:teuthology.orchestra.run.smithi087.stderr:sudo: ntpd: command not found 2023-12-24T17:53:29.505 INFO:teuthology.orchestra.run.smithi087.stdout:506 Cannot talk to daemon 2023-12-24T17:53:29.519 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-24T17:53:29.532 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-24T17:53:29.593 INFO:teuthology.orchestra.run.smithi087.stderr:bash: ntpq: command not found 2023-12-24T17:53:29.597 INFO:teuthology.orchestra.run.smithi087.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-24T17:53:29.597 INFO:teuthology.orchestra.run.smithi087.stdout:=============================================================================== 2023-12-24T17:53:29.597 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-24T17:53:29.597 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-24T17:53:29.597 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-24T17:53:29.597 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-24T17:53:29.598 INFO:teuthology.run_tasks:Running task install... 2023-12-24T17:53:29.606 DEBUG:teuthology.task.install:project ceph 2023-12-24T17:53:29.606 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f'}} 2023-12-24T17:53:29.606 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f'} 2023-12-24T17:53:29.606 INFO:teuthology.task.install:Using flavor: default 2023-12-24T17:53:29.611 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-24T17:53:29.611 INFO:teuthology.task.install:extra packages: [] 2023-12-24T17:53:29.612 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f', 'tag': None, 'wait_for_package': False} 2023-12-24T17:53:29.612 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f 2023-12-24T17:53:29.847 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-12-21-0917-quincy/5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f/centos/8/flavors/default/ 2023-12-24T17:53:29.847 INFO:teuthology.task.install.rpm:Package version is 17.2.7-168.g5e7ce129 2023-12-24T17:53:29.962 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-12-21-0917-quincy/5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-12-21-0917-quincy/5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-12-21-0917-quincy/5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-24T17:53:29.962 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-24T17:53:29.962 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-24T17:53:29.991 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-24T17:53:29.991 DEBUG:teuthology.orchestra.run.smithi087:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-24T17:53:30.060 DEBUG:teuthology.orchestra.run.smithi087:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-24T17:53:30.137 DEBUG:teuthology.orchestra.run.smithi087:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-24T17:53:30.203 INFO:teuthology.orchestra.run.smithi087.stdout:check_obsoletes = 1 2023-12-24T17:53:30.205 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum clean all 2023-12-24T17:53:30.481 INFO:teuthology.orchestra.run.smithi087.stdout:76 files removed 2023-12-24T17:53:30.508 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-radosgw 2023-12-24T17:53:31.215 INFO:teuthology.orchestra.run.smithi087.stdout:ceph packages for x86_64 257 kB/s | 78 kB 00:00 2023-12-24T17:53:31.522 INFO:teuthology.orchestra.run.smithi087.stdout:ceph noarch packages 34 kB/s | 9.1 kB 00:00 2023-12-24T17:53:31.915 INFO:teuthology.orchestra.run.smithi087.stdout:ceph source packages 4.0 kB/s | 1.5 kB 00:00 2023-12-24T17:53:32.965 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - AppStream 34 MB/s | 35 MB 00:01 2023-12-24T17:53:40.982 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - BaseOS 34 MB/s | 57 MB 00:01 2023-12-24T17:53:49.316 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - Extras 91 kB/s | 18 kB 00:00 2023-12-24T17:53:49.799 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - Extras common packages 23 kB/s | 6.9 kB 00:00 2023-12-24T17:53:53.768 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - PowerTools 1.7 MB/s | 6.6 MB 00:03 2023-12-24T17:53:55.210 INFO:teuthology.orchestra.run.smithi087.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2023-12-24T17:53:56.680 INFO:teuthology.orchestra.run.smithi087.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2023-12-24T17:54:00.620 INFO:teuthology.orchestra.run.smithi087.stdout:lab-extras 384 kB/s | 24 kB 00:00 2023-12-24T17:54:05.306 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-24T17:54:05.309 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:54:05.309 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2023-12-24T17:54:05.309 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:54:05.309 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-24T17:54:05.309 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-radosgw x86_64 2:17.2.7-168.g5e7ce129.el8 ceph 17 M 2023-12-24T17:54:05.309 INFO:teuthology.orchestra.run.smithi087.stdout:Upgrading: 2023-12-24T17:54:05.309 INFO:teuthology.orchestra.run.smithi087.stdout: librados2 x86_64 2:17.2.7-168.g5e7ce129.el8 ceph 3.5 M 2023-12-24T17:54:05.309 INFO:teuthology.orchestra.run.smithi087.stdout: librbd1 x86_64 2:17.2.7-168.g5e7ce129.el8 ceph 3.6 M 2023-12-24T17:54:05.309 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2023-12-24T17:54:05.309 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-base x86_64 2:17.2.7-168.g5e7ce129.el8 ceph 5.2 M 2023-12-24T17:54:05.309 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-common x86_64 2:17.2.7-168.g5e7ce129.el8 ceph 23 M 2023-12-24T17:54:05.309 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-selinux x86_64 2:17.2.7-168.g5e7ce129.el8 ceph 24 k 2023-12-24T17:54:05.310 INFO:teuthology.orchestra.run.smithi087.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-12-24T17:54:05.310 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs2 x86_64 2:17.2.7-168.g5e7ce129.el8 ceph 796 k 2023-12-24T17:54:05.310 INFO:teuthology.orchestra.run.smithi087.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-12-24T17:54:05.310 INFO:teuthology.orchestra.run.smithi087.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-12-24T17:54:05.310 INFO:teuthology.orchestra.run.smithi087.stdout: libradosstriper1 x86_64 2:17.2.7-168.g5e7ce129.el8 ceph 506 k 2023-12-24T17:54:05.310 INFO:teuthology.orchestra.run.smithi087.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2023-12-24T17:54:05.310 INFO:teuthology.orchestra.run.smithi087.stdout: librgw2 x86_64 2:17.2.7-168.g5e7ce129.el8 ceph 5.6 M 2023-12-24T17:54:05.310 INFO:teuthology.orchestra.run.smithi087.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-12-24T17:54:05.310 INFO:teuthology.orchestra.run.smithi087.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-12-24T17:54:05.310 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-argparse x86_64 2:17.2.7-168.g5e7ce129.el8 ceph 45 k 2023-12-24T17:54:05.310 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-common x86_64 2:17.2.7-168.g5e7ce129.el8 ceph 115 k 2023-12-24T17:54:05.310 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cephfs x86_64 2:17.2.7-168.g5e7ce129.el8 ceph 190 k 2023-12-24T17:54:05.310 INFO:teuthology.orchestra.run.smithi087.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-12-24T17:54:05.310 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rados x86_64 2:17.2.7-168.g5e7ce129.el8 ceph 371 k 2023-12-24T17:54:05.311 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rbd x86_64 2:17.2.7-168.g5e7ce129.el8 ceph 358 k 2023-12-24T17:54:05.311 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rgw x86_64 2:17.2.7-168.g5e7ce129.el8 ceph 107 k 2023-12-24T17:54:05.311 INFO:teuthology.orchestra.run.smithi087.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-12-24T17:54:05.311 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:54:05.311 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-24T17:54:05.311 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:54:05.311 INFO:teuthology.orchestra.run.smithi087.stdout:Install 21 Packages 2023-12-24T17:54:05.311 INFO:teuthology.orchestra.run.smithi087.stdout:Upgrade 2 Packages 2023-12-24T17:54:05.311 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:54:05.311 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 64 M 2023-12-24T17:54:05.312 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-24T17:54:07.171 INFO:teuthology.orchestra.run.smithi087.stdout:(1/23): ceph-base-17.2.7-168.g5e7ce129.el8.x86_ 3.7 MB/s | 5.2 MB 00:01 2023-12-24T17:54:07.246 INFO:teuthology.orchestra.run.smithi087.stdout:(2/23): ceph-selinux-17.2.7-168.g5e7ce129.el8.x 324 kB/s | 24 kB 00:00 2023-12-24T17:54:07.380 INFO:teuthology.orchestra.run.smithi087.stdout:(3/23): libcephfs2-17.2.7-168.g5e7ce129.el8.x86 5.8 MB/s | 796 kB 00:00 2023-12-24T17:54:07.538 INFO:teuthology.orchestra.run.smithi087.stdout:(4/23): libradosstriper1-17.2.7-168.g5e7ce129.e 3.1 MB/s | 506 kB 00:00 2023-12-24T17:54:08.430 INFO:teuthology.orchestra.run.smithi087.stdout:(5/23): librgw2-17.2.7-168.g5e7ce129.el8.x86_64 6.2 MB/s | 5.6 MB 00:00 2023-12-24T17:54:08.621 INFO:teuthology.orchestra.run.smithi087.stdout:(6/23): ceph-radosgw-17.2.7-168.g5e7ce129.el8.x 5.9 MB/s | 17 MB 00:02 2023-12-24T17:54:08.656 INFO:teuthology.orchestra.run.smithi087.stdout:(7/23): python3-ceph-argparse-17.2.7-168.g5e7ce 198 kB/s | 45 kB 00:00 2023-12-24T17:54:08.673 INFO:teuthology.orchestra.run.smithi087.stdout:(8/23): python3-ceph-common-17.2.7-168.g5e7ce12 2.2 MB/s | 115 kB 00:00 2023-12-24T17:54:08.757 INFO:teuthology.orchestra.run.smithi087.stdout:(9/23): python3-cephfs-17.2.7-168.g5e7ce129.el8 1.9 MB/s | 190 kB 00:00 2023-12-24T17:54:08.782 INFO:teuthology.orchestra.run.smithi087.stdout:(10/23): python3-rados-17.2.7-168.g5e7ce129.el8 3.3 MB/s | 371 kB 00:00 2023-12-24T17:54:09.007 INFO:teuthology.orchestra.run.smithi087.stdout:(11/23): ceph-common-17.2.7-168.g5e7ce129.el8.x 7.2 MB/s | 23 MB 00:03 2023-12-24T17:54:09.032 INFO:teuthology.orchestra.run.smithi087.stdout:(12/23): python3-rgw-17.2.7-168.g5e7ce129.el8.x 427 kB/s | 107 kB 00:00 2023-12-24T17:54:09.125 INFO:teuthology.orchestra.run.smithi087.stdout:(13/23): python3-rbd-17.2.7-168.g5e7ce129.el8.x 975 kB/s | 358 kB 00:00 2023-12-24T17:54:09.200 INFO:teuthology.orchestra.run.smithi087.stdout:(14/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.5 MB/s | 259 kB 00:00 2023-12-24T17:54:09.225 INFO:teuthology.orchestra.run.smithi087.stdout:(15/23): librdkafka-1.6.1-1.el8.x86_64.rpm 3.0 MB/s | 677 kB 00:00 2023-12-24T17:54:09.250 INFO:teuthology.orchestra.run.smithi087.stdout:(16/23): python3-prettytable-0.7.2-14.el8.noarc 350 kB/s | 44 kB 00:00 2023-12-24T17:54:09.359 INFO:teuthology.orchestra.run.smithi087.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 544 kB/s | 59 kB 00:00 2023-12-24T17:54:09.392 INFO:teuthology.orchestra.run.smithi087.stdout:(18/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 244 kB/s | 47 kB 00:00 2023-12-24T17:54:09.434 INFO:teuthology.orchestra.run.smithi087.stdout:(19/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.4 MB/s | 306 kB 00:00 2023-12-24T17:54:09.468 INFO:teuthology.orchestra.run.smithi087.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 690 kB/s | 75 kB 00:00 2023-12-24T17:54:09.910 INFO:teuthology.orchestra.run.smithi087.stdout:(21/23): librbd1-17.2.7-168.g5e7ce129.el8.x86_6 8.1 MB/s | 3.6 MB 00:00 2023-12-24T17:54:10.037 INFO:teuthology.orchestra.run.smithi087.stdout:(22/23): thrift-0.13.0-2.el8.x86_64.rpm 2.7 MB/s | 1.7 MB 00:00 2023-12-24T17:54:10.460 INFO:teuthology.orchestra.run.smithi087.stdout:(23/23): librados2-17.2.7-168.g5e7ce129.el8.x86 3.4 MB/s | 3.5 MB 00:01 2023-12-24T17:54:10.462 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-24T17:54:10.463 INFO:teuthology.orchestra.run.smithi087.stdout:Total 12 MB/s | 64 MB 00:05 2023-12-24T17:54:10.508 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-24T17:54:10.554 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-24T17:54:10.554 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-24T17:54:10.913 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-24T17:54:10.916 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-24T17:54:12.070 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-24T17:54:12.447 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-12-24T17:54:12.625 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-12-24T17:54:13.006 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-12-24T17:54:13.200 INFO:teuthology.orchestra.run.smithi087.stdout: Upgrading : librados2-2:17.2.7-168.g5e7ce129.el8.x86_64 3/25 2023-12-24T17:54:13.286 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados2-2:17.2.7-168.g5e7ce129.el8.x86_64 3/25 2023-12-24T17:54:13.592 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rados-2:17.2.7-168.g5e7ce129.el8.x86_64 4/25 2023-12-24T17:54:13.800 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephfs2-2:17.2.7-168.g5e7ce129.el8.x86_64 5/25 2023-12-24T17:54:14.193 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephfs2-2:17.2.7-168.g5e7ce129.el8.x86_64 5/25 2023-12-24T17:54:14.400 INFO:teuthology.orchestra.run.smithi087.stdout: Upgrading : librbd1-2:17.2.7-168.g5e7ce129.el8.x86_64 6/25 2023-12-24T17:54:14.629 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librbd1-2:17.2.7-168.g5e7ce129.el8.x86_64 6/25 2023-12-24T17:54:14.830 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2023-12-24T17:54:15.043 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 8/25 2023-12-24T17:54:15.735 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/25 2023-12-24T17:54:15.883 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librgw2-2:17.2.7-168.g5e7ce129.el8.x86_64 10/25 2023-12-24T17:54:15.953 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librgw2-2:17.2.7-168.g5e7ce129.el8.x86_64 10/25 2023-12-24T17:54:16.152 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ceph-argparse-2:17.2.7-168.g5e7ce129.el8.x 11/25 2023-12-24T17:54:16.311 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cephfs-2:17.2.7-168.g5e7ce129.el8.x86_64 12/25 2023-12-24T17:54:16.517 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rgw-2:17.2.7-168.g5e7ce129.el8.x86_64 13/25 2023-12-24T17:54:16.747 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rbd-2:17.2.7-168.g5e7ce129.el8.x86_64 14/25 2023-12-24T17:54:16.891 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libradosstriper1-2:17.2.7-168.g5e7ce129.el8.x86_64 15/25 2023-12-24T17:54:16.966 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libradosstriper1-2:17.2.7-168.g5e7ce129.el8.x86_64 15/25 2023-12-24T17:54:17.184 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2023-12-24T17:54:17.367 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2023-12-24T17:54:17.557 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-12-24T17:54:17.984 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ceph-common-2:17.2.7-168.g5e7ce129.el8.x86 19/25 2023-12-24T17:54:20.628 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-common-2:17.2.7-168.g5e7ce129.el8.x86_64 20/25 2023-12-24T17:54:20.883 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-common-2:17.2.7-168.g5e7ce129.el8.x86_64 20/25 2023-12-24T17:54:21.460 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-common-2:17.2.7-168.g5e7ce129.el8.x86_64 20/25 2023-12-24T17:54:21.715 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-base-2:17.2.7-168.g5e7ce129.el8.x86_64 21/25 2023-12-24T17:54:21.827 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-base-2:17.2.7-168.g5e7ce129.el8.x86_64 21/25 2023-12-24T17:54:21.982 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-selinux-2:17.2.7-168.g5e7ce129.el8.x86_64 22/25 2023-12-24T17:54:40.764 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-selinux-2:17.2.7-168.g5e7ce129.el8.x86_64 22/25 2023-12-24T17:54:40.764 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /sys 2023-12-24T17:54:40.764 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /proc 2023-12-24T17:54:40.764 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /dev 2023-12-24T17:54:40.764 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /run 2023-12-24T17:54:40.764 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /mnt 2023-12-24T17:54:40.764 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /var/tmp 2023-12-24T17:54:40.764 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /home 2023-12-24T17:54:40.764 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /tmp 2023-12-24T17:54:40.765 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /dev 2023-12-24T17:54:40.765 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:54:42.479 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-radosgw-2:17.2.7-168.g5e7ce129.el8.x86_64 23/25 2023-12-24T17:54:42.555 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-168.g5e7ce129.el8.x86_64 23/25 2023-12-24T17:54:42.555 INFO:teuthology.orchestra.run.smithi087.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-12-24T17:54:42.782 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-12-24T17:54:42.782 INFO:teuthology.orchestra.run.smithi087.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-12-24T17:54:43.509 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-12-24T17:54:43.509 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-base-2:17.2.7-168.g5e7ce129.el8.x86_64 1/25 2023-12-24T17:54:43.509 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-common-2:17.2.7-168.g5e7ce129.el8.x86_64 2/25 2023-12-24T17:54:43.509 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-radosgw-2:17.2.7-168.g5e7ce129.el8.x86_64 3/25 2023-12-24T17:54:43.510 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-selinux-2:17.2.7-168.g5e7ce129.el8.x86_64 4/25 2023-12-24T17:54:43.510 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephfs2-2:17.2.7-168.g5e7ce129.el8.x86_64 5/25 2023-12-24T17:54:43.510 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libradosstriper1-2:17.2.7-168.g5e7ce129.el8.x86_64 6/25 2023-12-24T17:54:43.510 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librgw2-2:17.2.7-168.g5e7ce129.el8.x86_64 7/25 2023-12-24T17:54:43.510 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ceph-argparse-2:17.2.7-168.g5e7ce129.el8.x 8/25 2023-12-24T17:54:43.510 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ceph-common-2:17.2.7-168.g5e7ce129.el8.x86 9/25 2023-12-24T17:54:43.510 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cephfs-2:17.2.7-168.g5e7ce129.el8.x86_64 10/25 2023-12-24T17:54:43.510 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rados-2:17.2.7-168.g5e7ce129.el8.x86_64 11/25 2023-12-24T17:54:43.510 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rbd-2:17.2.7-168.g5e7ce129.el8.x86_64 12/25 2023-12-24T17:54:43.510 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rgw-2:17.2.7-168.g5e7ce129.el8.x86_64 13/25 2023-12-24T17:54:43.510 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2023-12-24T17:54:43.510 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-12-24T17:54:43.510 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-12-24T17:54:43.510 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-12-24T17:54:43.511 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-12-24T17:54:43.511 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-12-24T17:54:43.511 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-12-24T17:54:43.511 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-12-24T17:54:43.511 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados2-2:17.2.7-168.g5e7ce129.el8.x86_64 22/25 2023-12-24T17:54:43.511 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-12-24T17:54:43.511 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librbd1-2:17.2.7-168.g5e7ce129.el8.x86_64 24/25 2023-12-24T17:54:44.819 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-12-24T17:54:44.820 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:54:44.820 INFO:teuthology.orchestra.run.smithi087.stdout:Upgraded: 2023-12-24T17:54:44.820 INFO:teuthology.orchestra.run.smithi087.stdout: librados2-2:17.2.7-168.g5e7ce129.el8.x86_64 2023-12-24T17:54:44.820 INFO:teuthology.orchestra.run.smithi087.stdout: librbd1-2:17.2.7-168.g5e7ce129.el8.x86_64 2023-12-24T17:54:44.820 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-24T17:54:44.820 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-base-2:17.2.7-168.g5e7ce129.el8.x86_64 2023-12-24T17:54:44.820 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-common-2:17.2.7-168.g5e7ce129.el8.x86_64 2023-12-24T17:54:44.820 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-radosgw-2:17.2.7-168.g5e7ce129.el8.x86_64 2023-12-24T17:54:44.820 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-selinux-2:17.2.7-168.g5e7ce129.el8.x86_64 2023-12-24T17:54:44.820 INFO:teuthology.orchestra.run.smithi087.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-12-24T17:54:44.820 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs2-2:17.2.7-168.g5e7ce129.el8.x86_64 2023-12-24T17:54:44.820 INFO:teuthology.orchestra.run.smithi087.stdout: liboath-2.6.2-3.el8.x86_64 2023-12-24T17:54:44.820 INFO:teuthology.orchestra.run.smithi087.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-12-24T17:54:44.820 INFO:teuthology.orchestra.run.smithi087.stdout: libradosstriper1-2:17.2.7-168.g5e7ce129.el8.x86_64 2023-12-24T17:54:44.821 INFO:teuthology.orchestra.run.smithi087.stdout: librdkafka-1.6.1-1.el8.x86_64 2023-12-24T17:54:44.821 INFO:teuthology.orchestra.run.smithi087.stdout: librgw2-2:17.2.7-168.g5e7ce129.el8.x86_64 2023-12-24T17:54:44.821 INFO:teuthology.orchestra.run.smithi087.stdout: libunwind-1.3.1-3.el8.x86_64 2023-12-24T17:54:44.821 INFO:teuthology.orchestra.run.smithi087.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-12-24T17:54:44.821 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-argparse-2:17.2.7-168.g5e7ce129.el8.x86_64 2023-12-24T17:54:44.821 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-common-2:17.2.7-168.g5e7ce129.el8.x86_64 2023-12-24T17:54:44.821 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cephfs-2:17.2.7-168.g5e7ce129.el8.x86_64 2023-12-24T17:54:44.821 INFO:teuthology.orchestra.run.smithi087.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-12-24T17:54:44.821 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rados-2:17.2.7-168.g5e7ce129.el8.x86_64 2023-12-24T17:54:44.821 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rbd-2:17.2.7-168.g5e7ce129.el8.x86_64 2023-12-24T17:54:44.821 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rgw-2:17.2.7-168.g5e7ce129.el8.x86_64 2023-12-24T17:54:44.821 INFO:teuthology.orchestra.run.smithi087.stdout: thrift-0.13.0-2.el8.x86_64 2023-12-24T17:54:44.822 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:54:44.822 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-24T17:54:45.039 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-test 2023-12-24T17:54:45.883 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:45 ago on Sun 24 Dec 2023 05:54:00 PM UTC. 2023-12-24T17:54:47.564 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-24T17:54:47.565 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:54:47.565 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2023-12-24T17:54:47.565 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:54:47.566 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-24T17:54:47.566 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-test x86_64 2:17.2.7-168.g5e7ce129.el8 ceph 52 M 2023-12-24T17:54:47.566 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2023-12-24T17:54:47.566 INFO:teuthology.orchestra.run.smithi087.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-12-24T17:54:47.566 INFO:teuthology.orchestra.run.smithi087.stdout: libcephsqlite x86_64 2:17.2.7-168.g5e7ce129.el8 ceph 174 k 2023-12-24T17:54:47.566 INFO:teuthology.orchestra.run.smithi087.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-12-24T17:54:47.566 INFO:teuthology.orchestra.run.smithi087.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-12-24T17:54:47.566 INFO:teuthology.orchestra.run.smithi087.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-12-24T17:54:47.566 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:54:47.566 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-24T17:54:47.566 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:54:47.566 INFO:teuthology.orchestra.run.smithi087.stdout:Install 6 Packages 2023-12-24T17:54:47.566 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:54:47.567 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 53 M 2023-12-24T17:54:47.567 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 234 M 2023-12-24T17:54:47.567 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-24T17:54:47.937 INFO:teuthology.orchestra.run.smithi087.stdout:(1/6): jq-1.6-8.el8.x86_64.rpm 1.1 MB/s | 203 kB 00:00 2023-12-24T17:54:47.988 INFO:teuthology.orchestra.run.smithi087.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2023-12-24T17:54:48.037 INFO:teuthology.orchestra.run.smithi087.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 6.3 MB/s | 323 kB 00:00 2023-12-24T17:54:48.079 INFO:teuthology.orchestra.run.smithi087.stdout:(4/6): libcephsqlite-17.2.7-168.g5e7ce129.el8.x 549 kB/s | 174 kB 00:00 2023-12-24T17:54:48.129 INFO:teuthology.orchestra.run.smithi087.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 769 kB/s | 71 kB 00:00 2023-12-24T17:54:51.729 INFO:teuthology.orchestra.run.smithi087.stdout:(6/6): ceph-test-17.2.7-168.g5e7ce129.el8.x86_6 13 MB/s | 52 MB 00:03 2023-12-24T17:54:51.731 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-24T17:54:51.731 INFO:teuthology.orchestra.run.smithi087.stdout:Total 13 MB/s | 53 MB 00:04 2023-12-24T17:54:51.774 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-24T17:54:51.795 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-24T17:54:51.795 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-24T17:54:52.128 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-24T17:54:52.131 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-24T17:54:52.636 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-24T17:54:52.869 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-12-24T17:54:53.072 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-12-24T17:54:53.220 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-24T17:54:53.325 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-24T17:54:53.502 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-12-24T17:54:53.653 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephsqlite-2:17.2.7-168.g5e7ce129.el8.x86_64 5/6 2023-12-24T17:54:58.686 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephsqlite-2:17.2.7-168.g5e7ce129.el8.x86_64 5/6 2023-12-24T17:54:58.934 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-test-2:17.2.7-168.g5e7ce129.el8.x86_64 6/6 2023-12-24T17:54:59.230 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-test-2:17.2.7-168.g5e7ce129.el8.x86_64 6/6 2023-12-24T17:54:59.230 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-test-2:17.2.7-168.g5e7ce129.el8.x86_64 1/6 2023-12-24T17:54:59.230 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephsqlite-2:17.2.7-168.g5e7ce129.el8.x86_64 2/6 2023-12-24T17:54:59.230 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-12-24T17:54:59.230 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-12-24T17:54:59.230 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-12-24T17:54:59.978 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-12-24T17:54:59.978 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:54:59.978 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-24T17:54:59.979 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-test-2:17.2.7-168.g5e7ce129.el8.x86_64 2023-12-24T17:54:59.979 INFO:teuthology.orchestra.run.smithi087.stdout: jq-1.6-8.el8.x86_64 2023-12-24T17:54:59.979 INFO:teuthology.orchestra.run.smithi087.stdout: libcephsqlite-2:17.2.7-168.g5e7ce129.el8.x86_64 2023-12-24T17:54:59.979 INFO:teuthology.orchestra.run.smithi087.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-12-24T17:54:59.979 INFO:teuthology.orchestra.run.smithi087.stdout: socat-1.7.4.1-1.el8.x86_64 2023-12-24T17:54:59.979 INFO:teuthology.orchestra.run.smithi087.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-12-24T17:54:59.979 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:54:59.979 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-24T17:55:00.199 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph 2023-12-24T17:55:01.052 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:01 ago on Sun 24 Dec 2023 05:54:00 PM UTC. 2023-12-24T17:55:02.738 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-24T17:55:02.741 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:55:02.742 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2023-12-24T17:55:02.742 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:55:02.742 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-24T17:55:02.742 INFO:teuthology.orchestra.run.smithi087.stdout: ceph x86_64 2:17.2.7-168.g5e7ce129.el8 ceph 6.4 k 2023-12-24T17:55:02.742 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2023-12-24T17:55:02.742 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mds x86_64 2:17.2.7-168.g5e7ce129.el8 ceph 2.3 M 2023-12-24T17:55:02.742 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr x86_64 2:17.2.7-168.g5e7ce129.el8 ceph 1.6 M 2023-12-24T17:55:02.742 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-modules-core noarch 2:17.2.7-168.g5e7ce129.el8 ceph-noarch 238 k 2023-12-24T17:55:02.742 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mon x86_64 2:17.2.7-168.g5e7ce129.el8 ceph 3.9 M 2023-12-24T17:55:02.742 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-osd x86_64 2:17.2.7-168.g5e7ce129.el8 ceph 17 M 2023-12-24T17:55:02.742 INFO:teuthology.orchestra.run.smithi087.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-12-24T17:55:02.742 INFO:teuthology.orchestra.run.smithi087.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-12-24T17:55:02.742 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-12-24T17:55:02.742 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-12-24T17:55:02.742 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-12-24T17:55:02.743 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-12-24T17:55:02.743 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-12-24T17:55:02.743 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-12-24T17:55:02.743 INFO:teuthology.orchestra.run.smithi087.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-12-24T17:55:02.743 INFO:teuthology.orchestra.run.smithi087.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-12-24T17:55:02.743 INFO:teuthology.orchestra.run.smithi087.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-12-24T17:55:02.743 INFO:teuthology.orchestra.run.smithi087.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-12-24T17:55:02.743 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-12-24T17:55:02.743 INFO:teuthology.orchestra.run.smithi087.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-12-24T17:55:02.743 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-12-24T17:55:02.743 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-12-24T17:55:02.743 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-12-24T17:55:02.743 INFO:teuthology.orchestra.run.smithi087.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-12-24T17:55:02.743 INFO:teuthology.orchestra.run.smithi087.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-12-24T17:55:02.743 INFO:teuthology.orchestra.run.smithi087.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-12-24T17:55:02.744 INFO:teuthology.orchestra.run.smithi087.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-12-24T17:55:02.744 INFO:teuthology.orchestra.run.smithi087.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-12-24T17:55:02.744 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-12-24T17:55:02.744 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-12-24T17:55:02.744 INFO:teuthology.orchestra.run.smithi087.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-12-24T17:55:02.744 INFO:teuthology.orchestra.run.smithi087.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-12-24T17:55:02.744 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:55:02.744 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-24T17:55:02.744 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:55:02.744 INFO:teuthology.orchestra.run.smithi087.stdout:Install 32 Packages 2023-12-24T17:55:02.744 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:55:02.745 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 28 M 2023-12-24T17:55:02.745 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 110 M 2023-12-24T17:55:02.745 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-24T17:55:10.715 INFO:teuthology.orchestra.run.smithi087.stdout:(1/32): ceph-17.2.7-168.g5e7ce129.el8.x86_64.rp 52 kB/s | 6.4 kB 00:00 2023-12-24T17:55:10.940 INFO:teuthology.orchestra.run.smithi087.stdout:(2/32): ceph-mds-17.2.7-168.g5e7ce129.el8.x86_6 6.6 MB/s | 2.3 MB 00:00 2023-12-24T17:55:10.982 INFO:teuthology.orchestra.run.smithi087.stdout:(3/32): ceph-mgr-17.2.7-168.g5e7ce129.el8.x86_6 4.0 MB/s | 1.6 MB 00:00 2023-12-24T17:55:11.090 INFO:teuthology.orchestra.run.smithi087.stdout:(4/32): ceph-mgr-modules-core-17.2.7-168.g5e7ce 2.1 MB/s | 238 kB 00:00 2023-12-24T17:55:11.232 INFO:teuthology.orchestra.run.smithi087.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.1 MB/s | 157 kB 00:00 2023-12-24T17:55:11.282 INFO:teuthology.orchestra.run.smithi087.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 778 kB/s | 39 kB 00:00 2023-12-24T17:55:11.333 INFO:teuthology.orchestra.run.smithi087.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 2.0 MB/s | 103 kB 00:00 2023-12-24T17:55:11.383 INFO:teuthology.orchestra.run.smithi087.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 1.1 MB/s | 57 kB 00:00 2023-12-24T17:55:11.458 INFO:teuthology.orchestra.run.smithi087.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 6.0 MB/s | 457 kB 00:00 2023-12-24T17:55:11.825 INFO:teuthology.orchestra.run.smithi087.stdout:(10/32): ceph-osd-17.2.7-168.g5e7ce129.el8.x86_ 19 MB/s | 17 MB 00:00 2023-12-24T17:55:11.900 INFO:teuthology.orchestra.run.smithi087.stdout:(11/32): ceph-mon-17.2.7-168.g5e7ce129.el8.x86_ 3.3 MB/s | 3.9 MB 00:01 2023-12-24T17:55:11.926 INFO:teuthology.orchestra.run.smithi087.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 532 kB/s | 249 kB 00:00 2023-12-24T17:55:12.126 INFO:teuthology.orchestra.run.smithi087.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 484 kB/s | 109 kB 00:00 2023-12-24T17:55:12.151 INFO:teuthology.orchestra.run.smithi087.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 194 kB/s | 44 kB 00:00 2023-12-24T17:55:12.193 INFO:teuthology.orchestra.run.smithi087.stdout:(15/32): python3-cryptography-3.2.1-6.el8.x86_6 1.7 MB/s | 641 kB 00:00 2023-12-24T17:55:12.318 INFO:teuthology.orchestra.run.smithi087.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.0 MB/s | 173 kB 00:00 2023-12-24T17:55:12.411 INFO:teuthology.orchestra.run.smithi087.stdout:(17/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 651 kB/s | 185 kB 00:00 2023-12-24T17:55:12.463 INFO:teuthology.orchestra.run.smithi087.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 77 kB/s | 11 kB 00:00 2023-12-24T17:55:12.511 INFO:teuthology.orchestra.run.smithi087.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 175 kB/s | 18 kB 00:00 2023-12-24T17:55:12.545 INFO:teuthology.orchestra.run.smithi087.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 600 kB/s | 49 kB 00:00 2023-12-24T17:55:12.570 INFO:teuthology.orchestra.run.smithi087.stdout:(21/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.0 MB/s | 384 kB 00:00 2023-12-24T17:55:12.604 INFO:teuthology.orchestra.run.smithi087.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 638 kB/s | 59 kB 00:00 2023-12-24T17:55:12.662 INFO:teuthology.orchestra.run.smithi087.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 169 kB/s | 16 kB 00:00 2023-12-24T17:55:12.704 INFO:teuthology.orchestra.run.smithi087.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 187 kB/s | 19 kB 00:00 2023-12-24T17:55:12.730 INFO:teuthology.orchestra.run.smithi087.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.5 MB/s | 283 kB 00:00 2023-12-24T17:55:12.771 INFO:teuthology.orchestra.run.smithi087.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 220 kB/s | 24 kB 00:00 2023-12-24T17:55:12.796 INFO:teuthology.orchestra.run.smithi087.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 317 kB/s | 29 kB 00:00 2023-12-24T17:55:12.822 INFO:teuthology.orchestra.run.smithi087.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 299 kB/s | 27 kB 00:00 2023-12-24T17:55:12.880 INFO:teuthology.orchestra.run.smithi087.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.3 MB/s | 255 kB 00:00 2023-12-24T17:55:12.955 INFO:teuthology.orchestra.run.smithi087.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 639 kB/s | 85 kB 00:00 2023-12-24T17:55:12.981 INFO:teuthology.orchestra.run.smithi087.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.3 MB/s | 251 kB 00:00 2023-12-24T17:55:12.997 INFO:teuthology.orchestra.run.smithi087.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 197 kB/s | 23 kB 00:00 2023-12-24T17:55:12.999 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-24T17:55:12.999 INFO:teuthology.orchestra.run.smithi087.stdout:Total 2.8 MB/s | 28 MB 00:10 2023-12-24T17:55:13.081 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-24T17:55:13.118 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-24T17:55:13.118 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-24T17:55:13.420 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-24T17:55:13.423 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-24T17:55:14.559 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-24T17:55:14.788 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-12-24T17:55:15.015 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-12-24T17:55:15.240 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-12-24T17:55:15.457 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-12-24T17:55:16.330 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-12-24T17:55:16.578 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-12-24T17:55:16.737 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-12-24T17:55:16.964 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-12-24T17:55:17.174 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-12-24T17:55:17.427 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-12-24T17:55:17.662 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-12-24T17:55:17.923 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-12-24T17:55:18.302 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-12-24T17:55:18.544 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-12-24T17:55:18.733 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-12-24T17:55:18.975 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-12-24T17:55:19.175 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-24T17:55:19.516 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-12-24T17:55:19.730 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-12-24T17:55:19.969 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-12-24T17:55:20.155 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-12-24T17:55:20.459 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-12-24T17:55:20.689 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-12-24T17:55:20.950 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-12-24T17:55:21.255 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-12-24T17:55:21.588 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-12-24T17:55:22.221 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-168.g5e7ce129.el8.n 27/32 2023-12-24T17:55:22.423 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-2:17.2.7-168.g5e7ce129.el8.x86_64 28/32 2023-12-24T17:55:23.912 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-2:17.2.7-168.g5e7ce129.el8.x86_64 28/32 2023-12-24T17:55:24.107 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-osd-2:17.2.7-168.g5e7ce129.el8.x86_64 29/32 2023-12-24T17:55:24.499 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-osd-2:17.2.7-168.g5e7ce129.el8.x86_64 29/32 2023-12-24T17:55:24.691 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mon-2:17.2.7-168.g5e7ce129.el8.x86_64 30/32 2023-12-24T17:55:24.926 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mon-2:17.2.7-168.g5e7ce129.el8.x86_64 30/32 2023-12-24T17:55:25.099 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mds-2:17.2.7-168.g5e7ce129.el8.x86_64 31/32 2023-12-24T17:55:25.118 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mds-2:17.2.7-168.g5e7ce129.el8.x86_64 31/32 2023-12-24T17:55:25.240 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-2:17.2.7-168.g5e7ce129.el8.x86_64 32/32 2023-12-24T17:55:25.647 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-2:17.2.7-168.g5e7ce129.el8.x86_64 32/32 2023-12-24T17:55:25.647 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-2:17.2.7-168.g5e7ce129.el8.x86_64 1/32 2023-12-24T17:55:25.648 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mds-2:17.2.7-168.g5e7ce129.el8.x86_64 2/32 2023-12-24T17:55:25.648 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-2:17.2.7-168.g5e7ce129.el8.x86_64 3/32 2023-12-24T17:55:25.648 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mon-2:17.2.7-168.g5e7ce129.el8.x86_64 4/32 2023-12-24T17:55:25.648 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-osd-2:17.2.7-168.g5e7ce129.el8.x86_64 5/32 2023-12-24T17:55:25.648 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-168.g5e7ce129.el8.n 6/32 2023-12-24T17:55:25.648 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-12-24T17:55:25.648 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-12-24T17:55:25.648 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-12-24T17:55:25.648 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-12-24T17:55:25.648 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-12-24T17:55:25.648 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-12-24T17:55:25.648 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-12-24T17:55:25.648 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-12-24T17:55:25.648 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-12-24T17:55:25.648 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-12-24T17:55:25.649 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-24T17:55:25.649 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-12-24T17:55:25.649 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-12-24T17:55:25.649 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-12-24T17:55:25.649 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-12-24T17:55:25.649 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-12-24T17:55:25.649 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-12-24T17:55:25.649 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-12-24T17:55:25.649 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-12-24T17:55:25.649 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-12-24T17:55:25.649 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-12-24T17:55:25.650 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-12-24T17:55:25.650 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-12-24T17:55:25.650 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-12-24T17:55:25.650 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-12-24T17:55:26.901 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-12-24T17:55:26.901 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:55:26.901 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-24T17:55:26.901 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-2:17.2.7-168.g5e7ce129.el8.x86_64 2023-12-24T17:55:26.901 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mds-2:17.2.7-168.g5e7ce129.el8.x86_64 2023-12-24T17:55:26.901 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-2:17.2.7-168.g5e7ce129.el8.x86_64 2023-12-24T17:55:26.901 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-modules-core-2:17.2.7-168.g5e7ce129.el8.noarch 2023-12-24T17:55:26.901 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mon-2:17.2.7-168.g5e7ce129.el8.x86_64 2023-12-24T17:55:26.901 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-osd-2:17.2.7-168.g5e7ce129.el8.x86_64 2023-12-24T17:55:26.901 INFO:teuthology.orchestra.run.smithi087.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-12-24T17:55:26.901 INFO:teuthology.orchestra.run.smithi087.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-12-24T17:55:26.902 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-12-24T17:55:26.902 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-12-24T17:55:26.902 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-12-24T17:55:26.902 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-12-24T17:55:26.902 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-6.2-6.el8.noarch 2023-12-24T17:55:26.902 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-12-24T17:55:26.902 INFO:teuthology.orchestra.run.smithi087.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-12-24T17:55:26.902 INFO:teuthology.orchestra.run.smithi087.stdout: python3-mako-1.0.6-14.el8.noarch 2023-12-24T17:55:26.902 INFO:teuthology.orchestra.run.smithi087.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-12-24T17:55:26.902 INFO:teuthology.orchestra.run.smithi087.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-12-24T17:55:26.902 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-12-24T17:55:26.902 INFO:teuthology.orchestra.run.smithi087.stdout: python3-portend-2.6-1.el8.noarch 2023-12-24T17:55:26.902 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-12-24T17:55:26.903 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pycparser-2.14-14.el8.noarch 2023-12-24T17:55:26.903 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pytz-2017.2-11.el8.noarch 2023-12-24T17:55:26.903 INFO:teuthology.orchestra.run.smithi087.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-12-24T17:55:26.903 INFO:teuthology.orchestra.run.smithi087.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-12-24T17:55:26.903 INFO:teuthology.orchestra.run.smithi087.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-12-24T17:55:26.903 INFO:teuthology.orchestra.run.smithi087.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-12-24T17:55:26.903 INFO:teuthology.orchestra.run.smithi087.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-12-24T17:55:26.903 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-12-24T17:55:26.903 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-12-24T17:55:26.903 INFO:teuthology.orchestra.run.smithi087.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-12-24T17:55:26.903 INFO:teuthology.orchestra.run.smithi087.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-12-24T17:55:26.903 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:55:26.903 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-24T17:55:27.051 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-base 2023-12-24T17:55:27.905 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:27 ago on Sun 24 Dec 2023 05:54:00 PM UTC. 2023-12-24T17:55:29.365 INFO:teuthology.orchestra.run.smithi087.stdout:Package ceph-base-2:17.2.7-168.g5e7ce129.el8.x86_64 is already installed. 2023-12-24T17:55:29.591 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-24T17:55:29.592 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2023-12-24T17:55:29.592 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-24T17:55:29.685 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install cephadm 2023-12-24T17:55:30.534 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:30 ago on Sun 24 Dec 2023 05:54:00 PM UTC. 2023-12-24T17:55:32.212 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-24T17:55:32.212 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:55:32.212 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2023-12-24T17:55:32.212 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:55:32.212 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-24T17:55:32.212 INFO:teuthology.orchestra.run.smithi087.stdout: cephadm noarch 2:17.2.7-168.g5e7ce129.el8 ceph-noarch 80 k 2023-12-24T17:55:32.212 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:55:32.213 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-24T17:55:32.213 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:55:32.213 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2023-12-24T17:55:32.213 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:55:32.213 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 80 k 2023-12-24T17:55:32.213 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 364 k 2023-12-24T17:55:32.213 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-24T17:55:32.471 INFO:teuthology.orchestra.run.smithi087.stdout:cephadm-17.2.7-168.g5e7ce129.el8.noarch.rpm 314 kB/s | 80 kB 00:00 2023-12-24T17:55:32.471 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-24T17:55:32.471 INFO:teuthology.orchestra.run.smithi087.stdout:Total 312 kB/s | 80 kB 00:00 2023-12-24T17:55:32.471 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-24T17:55:32.478 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-24T17:55:32.478 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-24T17:55:32.480 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-24T17:55:32.483 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-24T17:55:32.530 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-24T17:55:32.974 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: cephadm-2:17.2.7-168.g5e7ce129.el8.noarch 1/1 2023-12-24T17:55:33.168 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : cephadm-2:17.2.7-168.g5e7ce129.el8.noarch 1/1 2023-12-24T17:55:33.389 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: cephadm-2:17.2.7-168.g5e7ce129.el8.noarch 1/1 2023-12-24T17:55:34.030 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : cephadm-2:17.2.7-168.g5e7ce129.el8.noarch 1/1 2023-12-24T17:55:34.030 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:55:34.030 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-24T17:55:34.030 INFO:teuthology.orchestra.run.smithi087.stdout: cephadm-2:17.2.7-168.g5e7ce129.el8.noarch 2023-12-24T17:55:34.030 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:55:34.030 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-24T17:55:34.239 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-immutable-object-cache 2023-12-24T17:55:35.102 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:35 ago on Sun 24 Dec 2023 05:54:00 PM UTC. 2023-12-24T17:55:36.794 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-24T17:55:36.795 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:55:36.795 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2023-12-24T17:55:36.795 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:55:36.795 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-24T17:55:36.795 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-168.g5e7ce129.el8 ceph 162 k 2023-12-24T17:55:36.795 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:55:36.795 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-24T17:55:36.795 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:55:36.795 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2023-12-24T17:55:36.796 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:55:36.796 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 162 k 2023-12-24T17:55:36.796 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 490 k 2023-12-24T17:55:36.796 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-24T17:55:37.090 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-immutable-object-cache-17.2.7-168.g5e7ce12 552 kB/s | 162 kB 00:00 2023-12-24T17:55:37.090 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-24T17:55:37.090 INFO:teuthology.orchestra.run.smithi087.stdout:Total 549 kB/s | 162 kB 00:00 2023-12-24T17:55:37.091 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-24T17:55:37.101 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-24T17:55:37.101 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-24T17:55:37.149 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-24T17:55:37.152 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-24T17:55:37.282 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-24T17:55:37.481 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-168.g5e7ce129.e 1/1 2023-12-24T17:55:37.888 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-168.g5e7ce129.e 1/1 2023-12-24T17:55:38.544 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-168.g5e7ce129.e 1/1 2023-12-24T17:55:38.544 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:55:38.544 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-24T17:55:38.544 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-immutable-object-cache-2:17.2.7-168.g5e7ce129.el8.x86_64 2023-12-24T17:55:38.544 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:55:38.544 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-24T17:55:38.788 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr 2023-12-24T17:55:39.643 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:39 ago on Sun 24 Dec 2023 05:54:00 PM UTC. 2023-12-24T17:55:41.086 INFO:teuthology.orchestra.run.smithi087.stdout:Package ceph-mgr-2:17.2.7-168.g5e7ce129.el8.x86_64 is already installed. 2023-12-24T17:55:41.311 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-24T17:55:41.312 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2023-12-24T17:55:41.312 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-24T17:55:41.415 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-dashboard 2023-12-24T17:55:42.258 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:42 ago on Sun 24 Dec 2023 05:54:00 PM UTC. 2023-12-24T17:55:43.948 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-24T17:55:43.949 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:55:43.949 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2023-12-24T17:55:43.949 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:55:43.949 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-24T17:55:43.949 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-dashboard noarch 2:17.2.7-168.g5e7ce129.el8 ceph-noarch 1.7 M 2023-12-24T17:55:43.949 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2023-12-24T17:55:43.949 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-grafana-dashboards noarch 2:17.2.7-168.g5e7ce129.el8 ceph-noarch 23 k 2023-12-24T17:55:43.949 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-prometheus-alerts noarch 2:17.2.7-168.g5e7ce129.el8 ceph-noarch 14 k 2023-12-24T17:55:43.950 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-12-24T17:55:43.950 INFO:teuthology.orchestra.run.smithi087.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-12-24T17:55:43.950 INFO:teuthology.orchestra.run.smithi087.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-12-24T17:55:43.950 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:55:43.950 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-24T17:55:43.950 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:55:43.950 INFO:teuthology.orchestra.run.smithi087.stdout:Install 6 Packages 2023-12-24T17:55:43.950 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:55:43.950 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 2.0 M 2023-12-24T17:55:43.951 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 8.0 M 2023-12-24T17:55:43.951 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-24T17:55:44.384 INFO:teuthology.orchestra.run.smithi087.stdout:(1/6): ceph-grafana-dashboards-17.2.7-168.g5e7c 110 kB/s | 23 kB 00:00 2023-12-24T17:55:44.517 INFO:teuthology.orchestra.run.smithi087.stdout:(2/6): ceph-prometheus-alerts-17.2.7-168.g5e7ce 42 kB/s | 14 kB 00:00 2023-12-24T17:55:44.567 INFO:teuthology.orchestra.run.smithi087.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 232 kB/s | 43 kB 00:00 2023-12-24T17:55:44.726 INFO:teuthology.orchestra.run.smithi087.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 161 kB/s | 34 kB 00:00 2023-12-24T17:55:44.851 INFO:teuthology.orchestra.run.smithi087.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 703 kB/s | 196 kB 00:00 2023-12-24T17:55:44.968 INFO:teuthology.orchestra.run.smithi087.stdout:(6/6): ceph-mgr-dashboard-17.2.7-168.g5e7ce129. 2.1 MB/s | 1.7 MB 00:00 2023-12-24T17:55:44.971 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-24T17:55:44.972 INFO:teuthology.orchestra.run.smithi087.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2023-12-24T17:55:44.983 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-24T17:55:44.992 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-24T17:55:44.993 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-24T17:55:45.077 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-24T17:55:45.080 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-24T17:55:45.369 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-24T17:55:45.552 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-12-24T17:55:45.797 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-12-24T17:55:45.966 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-12-24T17:55:46.094 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-168.g5e7ce129.el8.no 4/6 2023-12-24T17:55:46.466 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-168.g5e7ce129.el8.n 5/6 2023-12-24T17:55:46.703 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-168.g5e7ce129.el8.noarch 6/6 2023-12-24T17:55:46.986 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-168.g5e7ce129.el8.noarch 6/6 2023-12-24T17:55:46.987 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-168.g5e7ce129.el8.n 1/6 2023-12-24T17:55:46.987 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-168.g5e7ce129.el8.noarch 2/6 2023-12-24T17:55:46.987 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-168.g5e7ce129.el8.no 3/6 2023-12-24T17:55:46.987 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-12-24T17:55:46.987 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-12-24T17:55:47.681 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-12-24T17:55:47.681 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:55:47.681 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-24T17:55:47.681 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-grafana-dashboards-2:17.2.7-168.g5e7ce129.el8.noarch 2023-12-24T17:55:47.681 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-dashboard-2:17.2.7-168.g5e7ce129.el8.noarch 2023-12-24T17:55:47.681 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-prometheus-alerts-2:17.2.7-168.g5e7ce129.el8.noarch 2023-12-24T17:55:47.681 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-12-24T17:55:47.682 INFO:teuthology.orchestra.run.smithi087.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-12-24T17:55:47.682 INFO:teuthology.orchestra.run.smithi087.stdout: python3-routes-2.4.1-12.el8.noarch 2023-12-24T17:55:47.682 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:55:47.682 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-24T17:55:47.807 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-24T17:55:48.670 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:48 ago on Sun 24 Dec 2023 05:54:00 PM UTC. 2023-12-24T17:55:50.367 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-24T17:55:50.368 INFO:teuthology.orchestra.run.smithi087.stdout:============================================================================================ 2023-12-24T17:55:50.369 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2023-12-24T17:55:50.369 INFO:teuthology.orchestra.run.smithi087.stdout:============================================================================================ 2023-12-24T17:55:50.369 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-24T17:55:50.369 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-168.g5e7ce129.el8 ceph-noarch 7.4 M 2023-12-24T17:55:50.369 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2023-12-24T17:55:50.369 INFO:teuthology.orchestra.run.smithi087.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-12-24T17:55:50.369 INFO:teuthology.orchestra.run.smithi087.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-12-24T17:55:50.369 INFO:teuthology.orchestra.run.smithi087.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-12-24T17:55:50.369 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-12-24T17:55:50.369 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-12-24T17:55:50.369 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-12-24T17:55:50.369 INFO:teuthology.orchestra.run.smithi087.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2023-12-24T17:55:50.370 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:55:50.370 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-24T17:55:50.370 INFO:teuthology.orchestra.run.smithi087.stdout:============================================================================================ 2023-12-24T17:55:50.370 INFO:teuthology.orchestra.run.smithi087.stdout:Install 8 Packages 2023-12-24T17:55:50.370 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:55:50.370 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 36 M 2023-12-24T17:55:50.370 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 210 M 2023-12-24T17:55:50.370 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-24T17:55:50.942 INFO:teuthology.orchestra.run.smithi087.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 14 MB/s | 4.7 MB 00:00 2023-12-24T17:55:51.017 INFO:teuthology.orchestra.run.smithi087.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 12 MB/s | 4.9 MB 00:00 2023-12-24T17:55:51.067 INFO:teuthology.orchestra.run.smithi087.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 4.4 MB/s | 225 kB 00:00 2023-12-24T17:55:51.126 INFO:teuthology.orchestra.run.smithi087.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 20 MB/s | 3.7 MB 00:00 2023-12-24T17:55:51.409 INFO:teuthology.orchestra.run.smithi087.stdout:(5/8): python3-scipy-1.0.0-21.module_el8+762+77 41 MB/s | 14 MB 00:00 2023-12-24T17:55:51.576 INFO:teuthology.orchestra.run.smithi087.stdout:(6/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.4 MB/s | 645 kB 00:00 2023-12-24T17:55:51.635 INFO:teuthology.orchestra.run.smithi087.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 767 kB/s | 172 kB 00:00 2023-12-24T17:55:51.727 INFO:teuthology.orchestra.run.smithi087.stdout:(8/8): ceph-mgr-diskprediction-local-17.2.7-168 6.6 MB/s | 7.4 MB 00:01 2023-12-24T17:55:51.728 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-24T17:55:51.728 INFO:teuthology.orchestra.run.smithi087.stdout:Total 26 MB/s | 36 MB 00:01 2023-12-24T17:55:51.958 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-24T17:55:51.977 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-24T17:55:51.977 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-24T17:55:52.395 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-24T17:55:52.398 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-24T17:55:52.935 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-24T17:55:53.079 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-24T17:55:53.206 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-24T17:55:53.387 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-24T17:55:53.985 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-24T17:55:54.204 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-12-24T17:55:54.823 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-12-24T17:55:55.731 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-24T17:55:56.433 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-24T17:55:56.774 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-12-24T17:55:57.006 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-24T17:55:58.758 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-24T17:55:59.976 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2023-12-24T17:56:00.211 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-168.g5e7ce129 8/8 2023-12-24T17:56:00.524 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-168.g5e7ce129 8/8 2023-12-24T17:56:00.525 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-168.g5e7ce129 1/8 2023-12-24T17:56:00.525 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-12-24T17:56:00.525 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-12-24T17:56:00.525 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-12-24T17:56:00.525 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-12-24T17:56:00.525 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2023-12-24T17:56:00.525 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-12-24T17:56:01.243 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-12-24T17:56:01.243 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:56:01.243 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-24T17:56:01.243 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-diskprediction-local-2:17.2.7-168.g5e7ce129.el8.noarch 2023-12-24T17:56:01.243 INFO:teuthology.orchestra.run.smithi087.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-12-24T17:56:01.243 INFO:teuthology.orchestra.run.smithi087.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-12-24T17:56:01.243 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-0.3.15-4.el8.x86_64 2023-12-24T17:56:01.243 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-12-24T17:56:01.244 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-12-24T17:56:01.244 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-12-24T17:56:01.244 INFO:teuthology.orchestra.run.smithi087.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2023-12-24T17:56:01.244 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:56:01.244 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-24T17:56:01.371 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-rook 2023-12-24T17:56:02.227 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:02 ago on Sun 24 Dec 2023 05:54:00 PM UTC. 2023-12-24T17:56:03.896 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-24T17:56:03.898 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:56:03.898 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2023-12-24T17:56:03.898 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:56:03.898 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-24T17:56:03.899 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-rook noarch 2:17.2.7-168.g5e7ce129.el8 ceph-noarch 47 k 2023-12-24T17:56:03.899 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2023-12-24T17:56:03.899 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-12-24T17:56:03.899 INFO:teuthology.orchestra.run.smithi087.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-12-24T17:56:03.899 INFO:teuthology.orchestra.run.smithi087.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-12-24T17:56:03.899 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-12-24T17:56:03.899 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-12-24T17:56:03.899 INFO:teuthology.orchestra.run.smithi087.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-12-24T17:56:03.899 INFO:teuthology.orchestra.run.smithi087.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-12-24T17:56:03.899 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-12-24T17:56:03.899 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-12-24T17:56:03.900 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-12-24T17:56:03.900 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-12-24T17:56:03.900 INFO:teuthology.orchestra.run.smithi087.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-12-24T17:56:03.900 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:56:03.900 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-24T17:56:03.900 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:56:03.900 INFO:teuthology.orchestra.run.smithi087.stdout:Install 13 Packages 2023-12-24T17:56:03.900 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:56:03.901 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 1.9 M 2023-12-24T17:56:03.901 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 25 M 2023-12-24T17:56:03.902 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-24T17:56:04.574 INFO:teuthology.orchestra.run.smithi087.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 252 kB/s | 27 kB 00:00 2023-12-24T17:56:04.600 INFO:teuthology.orchestra.run.smithi087.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 153 kB/s | 20 kB 00:00 2023-12-24T17:56:04.667 INFO:teuthology.orchestra.run.smithi087.stdout:(3/13): ceph-mgr-rook-17.2.7-168.g5e7ce129.el8. 239 kB/s | 47 kB 00:00 2023-12-24T17:56:04.725 INFO:teuthology.orchestra.run.smithi087.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 840 kB/s | 126 kB 00:00 2023-12-24T17:56:04.751 INFO:teuthology.orchestra.run.smithi087.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 730 kB/s | 110 kB 00:00 2023-12-24T17:56:04.909 INFO:teuthology.orchestra.run.smithi087.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 232 kB/s | 43 kB 00:00 2023-12-24T17:56:04.951 INFO:teuthology.orchestra.run.smithi087.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 545 kB/s | 155 kB 00:00 2023-12-24T17:56:04.985 INFO:teuthology.orchestra.run.smithi087.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 141 kB/s | 33 kB 00:00 2023-12-24T17:56:05.172 INFO:teuthology.orchestra.run.smithi087.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 62 kB/s | 16 kB 00:00 2023-12-24T17:56:05.260 INFO:teuthology.orchestra.run.smithi087.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 263 kB/s | 81 kB 00:00 2023-12-24T17:56:05.336 INFO:teuthology.orchestra.run.smithi087.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 393 kB/s | 64 kB 00:00 2023-12-24T17:56:05.361 INFO:teuthology.orchestra.run.smithi087.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 612 kB/s | 61 kB 00:00 2023-12-24T17:56:05.444 INFO:teuthology.orchestra.run.smithi087.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.5 MB/s | 1.2 MB 00:00 2023-12-24T17:56:05.445 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-24T17:56:05.445 INFO:teuthology.orchestra.run.smithi087.stdout:Total 1.2 MB/s | 1.9 MB 00:01 2023-12-24T17:56:05.504 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-24T17:56:05.520 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-24T17:56:05.520 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-24T17:56:05.607 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-24T17:56:05.610 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-24T17:56:06.125 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-24T17:56:06.388 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-12-24T17:56:06.605 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-12-24T17:56:06.796 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-12-24T17:56:07.025 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-12-24T17:56:07.241 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-12-24T17:56:07.473 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-12-24T17:56:07.791 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-12-24T17:56:08.065 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-12-24T17:56:09.089 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-12-24T17:56:09.333 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-12-24T17:56:09.561 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-12-24T17:56:09.760 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-12-24T17:56:09.952 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-rook-2:17.2.7-168.g5e7ce129.el8.noarch 13/13 2023-12-24T17:56:10.238 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-168.g5e7ce129.el8.noarch 13/13 2023-12-24T17:56:10.239 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-rook-2:17.2.7-168.g5e7ce129.el8.noarch 1/13 2023-12-24T17:56:10.239 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-12-24T17:56:10.239 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-12-24T17:56:10.239 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-12-24T17:56:10.239 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-12-24T17:56:10.239 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-12-24T17:56:10.239 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-12-24T17:56:10.239 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-12-24T17:56:10.239 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-12-24T17:56:10.239 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-12-24T17:56:10.239 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-12-24T17:56:10.239 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-12-24T17:56:11.009 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-12-24T17:56:11.009 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:56:11.009 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-24T17:56:11.009 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-rook-2:17.2.7-168.g5e7ce129.el8.noarch 2023-12-24T17:56:11.009 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-12-24T17:56:11.009 INFO:teuthology.orchestra.run.smithi087.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-12-24T17:56:11.009 INFO:teuthology.orchestra.run.smithi087.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-12-24T17:56:11.009 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-12-24T17:56:11.009 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-12-24T17:56:11.009 INFO:teuthology.orchestra.run.smithi087.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-12-24T17:56:11.009 INFO:teuthology.orchestra.run.smithi087.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-12-24T17:56:11.009 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-12-24T17:56:11.009 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-12-24T17:56:11.010 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-12-24T17:56:11.010 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rsa-4.9-2.el8.noarch 2023-12-24T17:56:11.010 INFO:teuthology.orchestra.run.smithi087.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-12-24T17:56:11.010 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:56:11.010 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-24T17:56:11.151 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-cephadm 2023-12-24T17:56:12.008 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:12 ago on Sun 24 Dec 2023 05:54:00 PM UTC. 2023-12-24T17:56:13.684 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-24T17:56:13.685 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:56:13.685 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2023-12-24T17:56:13.685 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:56:13.685 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-24T17:56:13.685 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-cephadm noarch 2:17.2.7-168.g5e7ce129.el8 ceph-noarch 121 k 2023-12-24T17:56:13.685 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2023-12-24T17:56:13.685 INFO:teuthology.orchestra.run.smithi087.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-12-24T17:56:13.685 INFO:teuthology.orchestra.run.smithi087.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-12-24T17:56:13.685 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-12-24T17:56:13.685 INFO:teuthology.orchestra.run.smithi087.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-12-24T17:56:13.686 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:56:13.686 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-24T17:56:13.686 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:56:13.686 INFO:teuthology.orchestra.run.smithi087.stdout:Install 5 Packages 2023-12-24T17:56:13.686 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:56:13.686 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 6.0 M 2023-12-24T17:56:13.686 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 26 M 2023-12-24T17:56:13.687 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-24T17:56:14.050 INFO:teuthology.orchestra.run.smithi087.stdout:(1/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 2.3 MB/s | 538 kB 00:00 2023-12-24T17:56:14.194 INFO:teuthology.orchestra.run.smithi087.stdout:(2/5): python3-babel-2.5.1-7.el8.noarch.rpm 13 MB/s | 4.8 MB 00:00 2023-12-24T17:56:14.361 INFO:teuthology.orchestra.run.smithi087.stdout:(3/5): ceph-mgr-cephadm-17.2.7-168.g5e7ce129.el 224 kB/s | 121 kB 00:00 2023-12-24T17:56:14.411 INFO:teuthology.orchestra.run.smithi087.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 278 kB/s | 60 kB 00:00 2023-12-24T17:56:14.436 INFO:teuthology.orchestra.run.smithi087.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.2 MB/s | 476 kB 00:00 2023-12-24T17:56:14.436 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-24T17:56:14.437 INFO:teuthology.orchestra.run.smithi087.stdout:Total 7.9 MB/s | 6.0 MB 00:00 2023-12-24T17:56:14.489 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-24T17:56:14.499 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-24T17:56:14.500 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-24T17:56:14.606 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-24T17:56:14.609 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-24T17:56:14.924 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-24T17:56:15.272 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-12-24T17:56:16.249 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-12-24T17:56:16.790 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-12-24T17:56:17.080 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-12-24T17:56:17.262 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-168.g5e7ce129.el8.noarch 5/5 2023-12-24T17:56:17.578 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-168.g5e7ce129.el8.noarch 5/5 2023-12-24T17:56:17.579 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-168.g5e7ce129.el8.noarch 1/5 2023-12-24T17:56:17.579 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-12-24T17:56:17.579 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-12-24T17:56:17.579 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-12-24T17:56:18.269 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-12-24T17:56:18.269 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:56:18.269 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-24T17:56:18.269 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-cephadm-2:17.2.7-168.g5e7ce129.el8.noarch 2023-12-24T17:56:18.269 INFO:teuthology.orchestra.run.smithi087.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-12-24T17:56:18.270 INFO:teuthology.orchestra.run.smithi087.stdout: python3-babel-2.5.1-7.el8.noarch 2023-12-24T17:56:18.270 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-12-24T17:56:18.270 INFO:teuthology.orchestra.run.smithi087.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-12-24T17:56:18.270 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:56:18.270 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-24T17:56:18.413 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-fuse 2023-12-24T17:56:19.261 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:19 ago on Sun 24 Dec 2023 05:54:00 PM UTC. 2023-12-24T17:56:20.937 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-24T17:56:20.938 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:56:20.938 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2023-12-24T17:56:20.938 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:56:20.938 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-24T17:56:20.938 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-fuse x86_64 2:17.2.7-168.g5e7ce129.el8 ceph 875 k 2023-12-24T17:56:20.938 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2023-12-24T17:56:20.938 INFO:teuthology.orchestra.run.smithi087.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-12-24T17:56:20.938 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:56:20.938 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-24T17:56:20.938 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:56:20.938 INFO:teuthology.orchestra.run.smithi087.stdout:Install 2 Packages 2023-12-24T17:56:20.938 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:56:20.941 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 958 k 2023-12-24T17:56:20.941 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 3.0 M 2023-12-24T17:56:20.941 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-24T17:56:21.336 INFO:teuthology.orchestra.run.smithi087.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 406 kB/s | 83 kB 00:00 2023-12-24T17:56:21.644 INFO:teuthology.orchestra.run.smithi087.stdout:(2/2): ceph-fuse-17.2.7-168.g5e7ce129.el8.x86_6 1.7 MB/s | 875 kB 00:00 2023-12-24T17:56:21.644 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-24T17:56:21.645 INFO:teuthology.orchestra.run.smithi087.stdout:Total 1.3 MB/s | 958 kB 00:00 2023-12-24T17:56:21.656 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-24T17:56:21.676 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-24T17:56:21.677 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-24T17:56:21.763 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-24T17:56:21.766 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-24T17:56:21.935 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-24T17:56:22.275 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-12-24T17:56:22.483 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-fuse-2:17.2.7-168.g5e7ce129.el8.x86_64 2/2 2023-12-24T17:56:22.898 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-fuse-2:17.2.7-168.g5e7ce129.el8.x86_64 2/2 2023-12-24T17:56:22.899 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-fuse-2:17.2.7-168.g5e7ce129.el8.x86_64 1/2 2023-12-24T17:56:23.572 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-12-24T17:56:23.572 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:56:23.573 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-24T17:56:23.573 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-fuse-2:17.2.7-168.g5e7ce129.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-12-24T17:56:23.573 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:56:23.573 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-24T17:56:23.767 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-volume 2023-12-24T17:56:24.629 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:24 ago on Sun 24 Dec 2023 05:54:00 PM UTC. 2023-12-24T17:56:26.295 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-24T17:56:26.295 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:56:26.296 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2023-12-24T17:56:26.296 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:56:26.296 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-24T17:56:26.296 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-volume noarch 2:17.2.7-168.g5e7ce129.el8 ceph-noarch 267 k 2023-12-24T17:56:26.296 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:56:26.296 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-24T17:56:26.296 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:56:26.296 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2023-12-24T17:56:26.296 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:56:26.296 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 267 k 2023-12-24T17:56:26.296 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 1.3 M 2023-12-24T17:56:26.297 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-24T17:56:26.733 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-volume-17.2.7-168.g5e7ce129.el8.noarch.rpm 616 kB/s | 267 kB 00:00 2023-12-24T17:56:26.734 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-24T17:56:26.735 INFO:teuthology.orchestra.run.smithi087.stdout:Total 614 kB/s | 267 kB 00:00 2023-12-24T17:56:26.735 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-24T17:56:26.742 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-24T17:56:26.742 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-24T17:56:26.779 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-24T17:56:26.783 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-24T17:56:26.988 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-24T17:56:27.568 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-volume-2:17.2.7-168.g5e7ce129.el8.noarch 1/1 2023-12-24T17:56:28.360 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-volume-2:17.2.7-168.g5e7ce129.el8.noarch 1/1 2023-12-24T17:56:29.960 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-volume-2:17.2.7-168.g5e7ce129.el8.noarch 1/1 2023-12-24T17:56:29.960 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:56:29.960 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-24T17:56:29.960 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-volume-2:17.2.7-168.g5e7ce129.el8.noarch 2023-12-24T17:56:29.960 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:56:29.960 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-24T17:56:30.189 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librados-devel 2023-12-24T17:56:31.059 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:31 ago on Sun 24 Dec 2023 05:54:00 PM UTC. 2023-12-24T17:56:32.728 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-24T17:56:32.729 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:56:32.729 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2023-12-24T17:56:32.729 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:56:32.729 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-24T17:56:32.729 INFO:teuthology.orchestra.run.smithi087.stdout: librados-devel x86_64 2:17.2.7-168.g5e7ce129.el8 ceph 130 k 2023-12-24T17:56:32.729 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:56:32.729 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-24T17:56:32.729 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:56:32.729 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2023-12-24T17:56:32.730 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:56:32.730 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 130 k 2023-12-24T17:56:32.731 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 464 k 2023-12-24T17:56:32.731 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-24T17:56:32.938 INFO:teuthology.orchestra.run.smithi087.stdout:librados-devel-17.2.7-168.g5e7ce129.el8.x86_64. 624 kB/s | 130 kB 00:00 2023-12-24T17:56:32.939 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-24T17:56:32.939 INFO:teuthology.orchestra.run.smithi087.stdout:Total 619 kB/s | 130 kB 00:00 2023-12-24T17:56:32.939 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-24T17:56:32.944 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-24T17:56:32.944 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-24T17:56:32.994 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-24T17:56:32.997 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-24T17:56:33.117 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-24T17:56:33.423 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librados-devel-2:17.2.7-168.g5e7ce129.el8.x86_64 1/1 2023-12-24T17:56:33.718 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados-devel-2:17.2.7-168.g5e7ce129.el8.x86_64 1/1 2023-12-24T17:56:34.378 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados-devel-2:17.2.7-168.g5e7ce129.el8.x86_64 1/1 2023-12-24T17:56:34.378 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:56:34.378 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-24T17:56:34.378 INFO:teuthology.orchestra.run.smithi087.stdout: librados-devel-2:17.2.7-168.g5e7ce129.el8.x86_64 2023-12-24T17:56:34.378 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:56:34.378 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-24T17:56:34.583 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install libcephfs2 2023-12-24T17:56:35.447 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:35 ago on Sun 24 Dec 2023 05:54:00 PM UTC. 2023-12-24T17:56:36.896 INFO:teuthology.orchestra.run.smithi087.stdout:Package libcephfs2-2:17.2.7-168.g5e7ce129.el8.x86_64 is already installed. 2023-12-24T17:56:37.122 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-24T17:56:37.124 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2023-12-24T17:56:37.124 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-24T17:56:37.246 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install libcephfs-devel 2023-12-24T17:56:38.097 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:38 ago on Sun 24 Dec 2023 05:54:00 PM UTC. 2023-12-24T17:56:39.785 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-24T17:56:39.785 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:56:39.785 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2023-12-24T17:56:39.785 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:56:39.785 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-24T17:56:39.785 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs-devel x86_64 2:17.2.7-168.g5e7ce129.el8 ceph 25 k 2023-12-24T17:56:39.785 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:56:39.786 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-24T17:56:39.786 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:56:39.786 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2023-12-24T17:56:39.786 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:56:39.786 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 25 k 2023-12-24T17:56:39.786 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 109 k 2023-12-24T17:56:39.786 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-24T17:56:40.007 INFO:teuthology.orchestra.run.smithi087.stdout:libcephfs-devel-17.2.7-168.g5e7ce129.el8.x86_64 113 kB/s | 25 kB 00:00 2023-12-24T17:56:40.008 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-24T17:56:40.008 INFO:teuthology.orchestra.run.smithi087.stdout:Total 112 kB/s | 25 kB 00:00 2023-12-24T17:56:40.008 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-24T17:56:40.011 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-24T17:56:40.011 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-24T17:56:40.013 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-24T17:56:40.016 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-24T17:56:40.066 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-24T17:56:40.279 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephfs-devel-2:17.2.7-168.g5e7ce129.el8.x86_64 1/1 2023-12-24T17:56:40.529 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-168.g5e7ce129.el8.x86_64 1/1 2023-12-24T17:56:41.330 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephfs-devel-2:17.2.7-168.g5e7ce129.el8.x86_64 1/1 2023-12-24T17:56:41.330 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:56:41.330 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-24T17:56:41.330 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs-devel-2:17.2.7-168.g5e7ce129.el8.x86_64 2023-12-24T17:56:41.331 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:56:41.331 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-24T17:56:41.466 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librados2 2023-12-24T17:56:42.320 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:42 ago on Sun 24 Dec 2023 05:54:00 PM UTC. 2023-12-24T17:56:43.767 INFO:teuthology.orchestra.run.smithi087.stdout:Package librados2-2:17.2.7-168.g5e7ce129.el8.x86_64 is already installed. 2023-12-24T17:56:43.993 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-24T17:56:43.994 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2023-12-24T17:56:43.994 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-24T17:56:44.084 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librbd1 2023-12-24T17:56:44.951 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:44 ago on Sun 24 Dec 2023 05:54:00 PM UTC. 2023-12-24T17:56:46.418 INFO:teuthology.orchestra.run.smithi087.stdout:Package librbd1-2:17.2.7-168.g5e7ce129.el8.x86_64 is already installed. 2023-12-24T17:56:46.643 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-24T17:56:46.645 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2023-12-24T17:56:46.645 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-24T17:56:46.748 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rados 2023-12-24T17:56:47.606 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:47 ago on Sun 24 Dec 2023 05:54:00 PM UTC. 2023-12-24T17:56:49.074 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rados-2:17.2.7-168.g5e7ce129.el8.x86_64 is already installed. 2023-12-24T17:56:49.306 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-24T17:56:49.308 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2023-12-24T17:56:49.308 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-24T17:56:49.404 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rgw 2023-12-24T17:56:50.269 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:50 ago on Sun 24 Dec 2023 05:54:00 PM UTC. 2023-12-24T17:56:51.718 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rgw-2:17.2.7-168.g5e7ce129.el8.x86_64 is already installed. 2023-12-24T17:56:51.944 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-24T17:56:51.945 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2023-12-24T17:56:51.946 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-24T17:56:52.050 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-cephfs 2023-12-24T17:56:52.905 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:52 ago on Sun 24 Dec 2023 05:54:00 PM UTC. 2023-12-24T17:56:54.365 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-cephfs-2:17.2.7-168.g5e7ce129.el8.x86_64 is already installed. 2023-12-24T17:56:54.591 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-24T17:56:54.592 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2023-12-24T17:56:54.592 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-24T17:56:54.691 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rbd 2023-12-24T17:56:55.545 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:55 ago on Sun 24 Dec 2023 05:54:00 PM UTC. 2023-12-24T17:56:56.998 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rbd-2:17.2.7-168.g5e7ce129.el8.x86_64 is already installed. 2023-12-24T17:56:57.225 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-24T17:56:57.226 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2023-12-24T17:56:57.226 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-24T17:56:57.324 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-fuse 2023-12-24T17:56:58.178 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:58 ago on Sun 24 Dec 2023 05:54:00 PM UTC. 2023-12-24T17:56:59.866 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-24T17:56:59.867 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:56:59.867 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2023-12-24T17:56:59.867 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:56:59.867 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-24T17:56:59.867 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-fuse x86_64 2:17.2.7-168.g5e7ce129.el8 ceph 90 k 2023-12-24T17:56:59.867 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:56:59.867 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-24T17:56:59.867 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:56:59.867 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2023-12-24T17:56:59.868 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:56:59.868 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 90 k 2023-12-24T17:56:59.868 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 264 k 2023-12-24T17:56:59.868 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-24T17:57:00.063 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-fuse-17.2.7-168.g5e7ce129.el8.x86_64.rpm 464 kB/s | 90 kB 00:00 2023-12-24T17:57:00.064 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-24T17:57:00.064 INFO:teuthology.orchestra.run.smithi087.stdout:Total 461 kB/s | 90 kB 00:00 2023-12-24T17:57:00.064 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-24T17:57:00.074 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-24T17:57:00.074 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-24T17:57:00.122 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-24T17:57:00.125 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-24T17:57:00.224 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-24T17:57:00.446 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-fuse-2:17.2.7-168.g5e7ce129.el8.x86_64 1/1 2023-12-24T17:57:00.744 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-fuse-2:17.2.7-168.g5e7ce129.el8.x86_64 1/1 2023-12-24T17:57:01.442 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-fuse-2:17.2.7-168.g5e7ce129.el8.x86_64 1/1 2023-12-24T17:57:01.488 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:01.488 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-24T17:57:01.489 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-fuse-2:17.2.7-168.g5e7ce129.el8.x86_64 2023-12-24T17:57:01.489 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:01.489 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-24T17:57:01.947 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-mirror 2023-12-24T17:57:02.804 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:03:02 ago on Sun 24 Dec 2023 05:54:00 PM UTC. 2023-12-24T17:57:04.490 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-24T17:57:04.491 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:57:04.491 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2023-12-24T17:57:04.491 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:57:04.491 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-24T17:57:04.491 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-mirror x86_64 2:17.2.7-168.g5e7ce129.el8 ceph 3.5 M 2023-12-24T17:57:04.491 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:04.491 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-24T17:57:04.491 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:57:04.491 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2023-12-24T17:57:04.491 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:04.491 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 3.5 M 2023-12-24T17:57:04.491 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 16 M 2023-12-24T17:57:04.491 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-24T17:57:05.374 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-mirror-17.2.7-168.g5e7ce129.el8.x86_64.rpm 3.9 MB/s | 3.5 MB 00:00 2023-12-24T17:57:05.374 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-24T17:57:05.375 INFO:teuthology.orchestra.run.smithi087.stdout:Total 3.9 MB/s | 3.5 MB 00:00 2023-12-24T17:57:05.375 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-24T17:57:05.387 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-24T17:57:05.387 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-24T17:57:05.451 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-24T17:57:05.454 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-24T17:57:05.888 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-24T17:57:06.133 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-mirror-2:17.2.7-168.g5e7ce129.el8.x86_64 1/1 2023-12-24T17:57:06.551 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-mirror-2:17.2.7-168.g5e7ce129.el8.x86_64 1/1 2023-12-24T17:57:07.250 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-mirror-2:17.2.7-168.g5e7ce129.el8.x86_64 1/1 2023-12-24T17:57:07.250 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:07.251 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-24T17:57:07.251 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-mirror-2:17.2.7-168.g5e7ce129.el8.x86_64 2023-12-24T17:57:07.251 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:07.251 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-24T17:57:07.455 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-nbd 2023-12-24T17:57:08.316 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:03:08 ago on Sun 24 Dec 2023 05:54:00 PM UTC. 2023-12-24T17:57:10.004 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-24T17:57:10.005 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:57:10.006 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2023-12-24T17:57:10.006 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:57:10.006 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-24T17:57:10.006 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-nbd x86_64 2:17.2.7-168.g5e7ce129.el8 ceph 175 k 2023-12-24T17:57:10.006 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:10.006 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-24T17:57:10.007 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-24T17:57:10.007 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2023-12-24T17:57:10.007 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:10.007 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 175 k 2023-12-24T17:57:10.007 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 556 k 2023-12-24T17:57:10.007 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-24T17:57:10.326 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-nbd-17.2.7-168.g5e7ce129.el8.x86_64.rpm 545 kB/s | 175 kB 00:00 2023-12-24T17:57:10.327 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-24T17:57:10.327 INFO:teuthology.orchestra.run.smithi087.stdout:Total 543 kB/s | 175 kB 00:00 2023-12-24T17:57:10.327 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-24T17:57:10.338 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-24T17:57:10.338 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-24T17:57:10.387 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-24T17:57:10.390 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-24T17:57:10.499 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-24T17:57:10.695 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-nbd-2:17.2.7-168.g5e7ce129.el8.x86_64 1/1 2023-12-24T17:57:11.018 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-nbd-2:17.2.7-168.g5e7ce129.el8.x86_64 1/1 2023-12-24T17:57:11.699 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-nbd-2:17.2.7-168.g5e7ce129.el8.x86_64 1/1 2023-12-24T17:57:11.699 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:11.699 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-24T17:57:11.699 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-nbd-2:17.2.7-168.g5e7ce129.el8.x86_64 2023-12-24T17:57:11.699 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:11.699 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-24T17:57:11.918 DEBUG:teuthology.parallel:result is None 2023-12-24T17:57:11.919 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f 2023-12-24T17:57:12.224 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-24T17:57:12.247 INFO:teuthology.orchestra.run.smithi087.stdout:17.2.7-168.g5e7ce129.el8 2023-12-24T17:57:12.248 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-168.g5e7ce129.el8 2023-12-24T17:57:12.248 INFO:teuthology.task.install:The correct ceph version 17.2.7-168.g5e7ce129 is installed. 2023-12-24T17:57:12.249 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-24T17:57:12.250 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-24T17:57:12.250 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-24T17:57:12.316 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-24T17:57:12.316 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-24T17:57:12.316 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/daemon-helper 2023-12-24T17:57:12.383 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-24T17:57:12.450 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-24T17:57:12.450 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-24T17:57:12.450 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-24T17:57:12.516 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-24T17:57:12.583 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-24T17:57:12.583 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-24T17:57:12.583 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/stdin-killer 2023-12-24T17:57:12.649 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-24T17:57:12.717 INFO:teuthology.run_tasks:Running task ceph... 2023-12-24T17:57:12.782 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-24T17:57:12.782 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 777 /var/log/ceph 2023-12-24T17:57:12.809 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-24T17:57:12.809 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-24T17:57:12.876 INFO:tasks.ceph:Creating extra log directories... 2023-12-24T17:57:12.876 DEBUG:teuthology.orchestra.run.smithi087:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-24T17:57:12.951 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-12-24T17:57:12.951 INFO:tasks.ceph:config {'conf': {'client': {'rbd default format': 1}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(CACHE_POOL_NO_HIT_SET\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-24T17:57:12.951 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-12-24_15:40:05-rbd-wip-yuri4-testing-2023-12-21-0917-quincy-distro-default-smithi/7500245', 'branch': 'wip-yuri4-testing-2023-12-21-0917-quincy', 'description': 'rbd/cli_v1/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/format-1 msgr-failures/few objectstore/bluestore-stupid pool/small-cache-pool supported-random-distro$/{centos_8} workloads/rbd_cli_generic}', 'email': None, 'first_in_suite': False, 'job_id': '7500245', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-12-24_15:40:05-rbd-wip-yuri4-testing-2023-12-21-0917-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri4-testing-2023-12-21-0917-quincy'}, 'ceph': {'conf': {'client': {'rbd default format': 1}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(CACHE_POOL_NO_HIT_SET\\)'], 'sha1': '5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri4-testing-2023-12-21-0917-quincy', 'sha1': '5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 1236, 'sha1': '5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-yuri4-testing-2023-12-21-0917-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f', 'targets': {'smithi087.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC3zm7LNATb0NkKyv/ek111PQVxF5d3AKbhUg0JJ5rq0jQEgeD6U4YLaQhyKi/4V9SdDJvED3VWEN53UE54teL0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'exec': {'client.0': ['sudo ceph osd pool create cache 4', 'sudo ceph osd tier add rbd cache', 'sudo ceph osd tier cache-mode cache writeback', 'sudo ceph osd tier set-overlay rbd cache', 'sudo ceph osd pool set cache hit_set_type bloom', 'sudo ceph osd pool set cache hit_set_count 8', 'sudo ceph osd pool set cache hit_set_period 60', 'sudo ceph osd pool set cache target_max_objects 250']}}, {'workunit': {'clients': {'client.0': ['rbd/cli_generic.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9680d2aba1e76f64ade295c81599c97907457848', 'timestamp': '2023-12-24_15:40:05', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078'} 2023-12-24T17:57:12.952 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-24T17:57:13.001 DEBUG:teuthology.orchestra.run.smithi087:> sudo install -d -m0777 -- /var/run/ceph 2023-12-24T17:57:13.068 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-24T17:57:13.068 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/scratch_devs of=/dev/stdout 2023-12-24T17:57:13.126 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-24T17:57:13.126 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_1 2023-12-24T17:57:13.184 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-24T17:57:13.184 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-24T17:57:13.184 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 6h/6d Inode: 397602 Links: 1 2023-12-24T17:57:13.184 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-24T17:57:13.184 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2023-12-24T17:57:13.184 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2023-12-24 17:57:06.301172076 +0000 2023-12-24T17:57:13.184 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2023-12-24 17:52:52.438733068 +0000 2023-12-24T17:57:13.184 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2023-12-24 17:52:52.438733068 +0000 2023-12-24T17:57:13.184 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: - 2023-12-24T17:57:13.185 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-24T17:57:13.250 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2023-12-24T17:57:13.250 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2023-12-24T17:57:13.250 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000228922 s, 2.2 MB/s 2023-12-24T17:57:13.251 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-24T17:57:13.310 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_2 2023-12-24T17:57:13.368 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-24T17:57:13.369 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-24T17:57:13.369 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 6h/6d Inode: 393717 Links: 1 2023-12-24T17:57:13.369 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-24T17:57:13.369 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2023-12-24T17:57:13.369 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2023-12-24 17:57:06.301172076 +0000 2023-12-24T17:57:13.369 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2023-12-24 17:52:52.933721906 +0000 2023-12-24T17:57:13.369 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2023-12-24 17:52:52.933721906 +0000 2023-12-24T17:57:13.369 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: - 2023-12-24T17:57:13.369 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-24T17:57:13.434 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2023-12-24T17:57:13.434 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2023-12-24T17:57:13.434 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000215257 s, 2.4 MB/s 2023-12-24T17:57:13.435 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-24T17:57:13.494 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_3 2023-12-24T17:57:13.552 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-24T17:57:13.552 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-24T17:57:13.552 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 6h/6d Inode: 391771 Links: 1 2023-12-24T17:57:13.552 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-24T17:57:13.552 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2023-12-24T17:57:13.553 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2023-12-24 17:57:06.301172076 +0000 2023-12-24T17:57:13.553 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2023-12-24 17:52:53.429710721 +0000 2023-12-24T17:57:13.553 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2023-12-24 17:52:53.429710721 +0000 2023-12-24T17:57:13.553 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: - 2023-12-24T17:57:13.553 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-24T17:57:13.618 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2023-12-24T17:57:13.618 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2023-12-24T17:57:13.618 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000225921 s, 2.3 MB/s 2023-12-24T17:57:13.619 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-24T17:57:13.678 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_4 2023-12-24T17:57:13.736 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-24T17:57:13.736 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-24T17:57:13.736 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 6h/6d Inode: 391845 Links: 1 2023-12-24T17:57:13.736 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-24T17:57:13.736 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2023-12-24T17:57:13.736 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2023-12-24 17:57:06.301172076 +0000 2023-12-24T17:57:13.736 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2023-12-24 17:52:53.943699131 +0000 2023-12-24T17:57:13.736 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2023-12-24 17:52:53.943699131 +0000 2023-12-24T17:57:13.736 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: - 2023-12-24T17:57:13.737 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-24T17:57:13.801 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2023-12-24T17:57:13.802 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2023-12-24T17:57:13.802 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000255639 s, 2.0 MB/s 2023-12-24T17:57:13.803 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-24T17:57:13.862 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-12-24T17:57:13.862 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi087.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-24T17:57:13.862 INFO:tasks.ceph:Generating config... 2023-12-24T17:57:13.863 INFO:tasks.ceph:[client] rbd default format = 1 2023-12-24T17:57:13.863 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-12-24T17:57:13.863 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-12-24T17:57:13.863 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-12-24T17:57:13.863 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-24T17:57:13.863 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-24T17:57:13.863 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-24T17:57:13.863 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-24T17:57:13.863 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-24T17:57:13.864 INFO:tasks.ceph:[osd] bdev async discard = True 2023-12-24T17:57:13.864 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-12-24T17:57:13.864 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2023-12-24T17:57:13.864 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-24T17:57:13.864 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-24T17:57:13.864 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-12-24T17:57:13.864 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-12-24T17:57:13.864 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-24T17:57:13.864 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-24T17:57:13.864 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-12-24T17:57:13.864 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-24T17:57:13.864 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-24T17:57:13.864 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-24T17:57:13.864 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-24T17:57:13.864 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-24T17:57:13.864 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-12-24T17:57:13.865 INFO:tasks.ceph:Setting up mon.a... 2023-12-24T17:57:13.865 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-12-24T17:57:13.949 INFO:teuthology.orchestra.run.smithi087.stdout:creating /etc/ceph/ceph.keyring 2023-12-24T17:57:13.954 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-12-24T17:57:13.994 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-24T17:57:14.061 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.87')] 2023-12-24T17:57:14.062 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.87', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'rbd default format': 1}, 'mon.a': {}} 2023-12-24T17:57:14.062 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-24T17:57:14.062 DEBUG:teuthology.orchestra.run.smithi087:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-24T17:57:14.119 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.87 --print /home/ubuntu/cephtest/ceph.monmap 2023-12-24T17:57:14.199 INFO:teuthology.orchestra.run.smithi087.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-12-24T17:57:14.200 INFO:teuthology.orchestra.run.smithi087.stdout:monmaptool: generated fsid 0f2278f5-9601-4df5-8109-682fbac125ff 2023-12-24T17:57:14.200 INFO:teuthology.orchestra.run.smithi087.stdout:setting min_mon_release = octopus 2023-12-24T17:57:14.200 INFO:teuthology.orchestra.run.smithi087.stdout:epoch 0 2023-12-24T17:57:14.200 INFO:teuthology.orchestra.run.smithi087.stdout:fsid 0f2278f5-9601-4df5-8109-682fbac125ff 2023-12-24T17:57:14.200 INFO:teuthology.orchestra.run.smithi087.stdout:last_changed 2023-12-24T17:57:14.203222+0000 2023-12-24T17:57:14.200 INFO:teuthology.orchestra.run.smithi087.stdout:created 2023-12-24T17:57:14.203222+0000 2023-12-24T17:57:14.201 INFO:teuthology.orchestra.run.smithi087.stdout:min_mon_release 15 (octopus) 2023-12-24T17:57:14.201 INFO:teuthology.orchestra.run.smithi087.stdout:election_strategy: 1 2023-12-24T17:57:14.201 INFO:teuthology.orchestra.run.smithi087.stdout:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2023-12-24T17:57:14.201 INFO:teuthology.orchestra.run.smithi087.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-12-24T17:57:14.203 DEBUG:teuthology.orchestra.run.smithi087:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-24T17:57:14.260 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 0f2278f5-9601-4df5-8109-682fbac125ff... 2023-12-24T17:57:14.261 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-24T17:57:14.344 INFO:teuthology.orchestra.run.smithi087.stdout:[global] 2023-12-24T17:57:14.344 INFO:teuthology.orchestra.run.smithi087.stdout: chdir = "" 2023-12-24T17:57:14.344 INFO:teuthology.orchestra.run.smithi087.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-24T17:57:14.344 INFO:teuthology.orchestra.run.smithi087.stdout: auth supported = cephx 2023-12-24T17:57:14.344 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:14.344 INFO:teuthology.orchestra.run.smithi087.stdout: filestore xattr use omap = true 2023-12-24T17:57:14.344 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:14.344 INFO:teuthology.orchestra.run.smithi087.stdout: mon clock drift allowed = 1.000 2023-12-24T17:57:14.344 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:14.344 INFO:teuthology.orchestra.run.smithi087.stdout: osd crush chooseleaf type = 0 2023-12-24T17:57:14.344 INFO:teuthology.orchestra.run.smithi087.stdout: auth debug = true 2023-12-24T17:57:14.345 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:14.345 INFO:teuthology.orchestra.run.smithi087.stdout: ms die on old message = true 2023-12-24T17:57:14.345 INFO:teuthology.orchestra.run.smithi087.stdout: ms die on bug = true 2023-12-24T17:57:14.345 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:14.345 INFO:teuthology.orchestra.run.smithi087.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-24T17:57:14.345 INFO:teuthology.orchestra.run.smithi087.stdout: mon pg warn max object skew = 0 2023-12-24T17:57:14.345 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:14.345 INFO:teuthology.orchestra.run.smithi087.stdout: # disable pg_autoscaler by default for new pools 2023-12-24T17:57:14.345 INFO:teuthology.orchestra.run.smithi087.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-24T17:57:14.345 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:14.345 INFO:teuthology.orchestra.run.smithi087.stdout: osd pool default size = 2 2023-12-24T17:57:14.345 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:14.345 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd allow primary affinity = true 2023-12-24T17:57:14.345 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd allow pg remap = true 2023-12-24T17:57:14.346 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on legacy crush tunables = false 2023-12-24T17:57:14.346 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on crush straw calc version zero = false 2023-12-24T17:57:14.346 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on no sortbitwise = false 2023-12-24T17:57:14.346 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on osd down out interval zero = false 2023-12-24T17:57:14.346 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on too few osds = false 2023-12-24T17:57:14.346 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-24T17:57:14.346 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-24T17:57:14.346 INFO:teuthology.orchestra.run.smithi087.stdout: mon_allow_pool_size_one = true 2023-12-24T17:57:14.346 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:14.346 INFO:teuthology.orchestra.run.smithi087.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-24T17:57:14.346 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:14.346 INFO:teuthology.orchestra.run.smithi087.stdout: osd default data pool replay window = 5 2023-12-24T17:57:14.346 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:14.346 INFO:teuthology.orchestra.run.smithi087.stdout: mon allow pool delete = true 2023-12-24T17:57:14.347 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:14.347 INFO:teuthology.orchestra.run.smithi087.stdout: mon cluster log file level = debug 2023-12-24T17:57:14.347 INFO:teuthology.orchestra.run.smithi087.stdout: debug asserts on shutdown = true 2023-12-24T17:57:14.347 INFO:teuthology.orchestra.run.smithi087.stdout: mon health detail to clog = false 2023-12-24T17:57:14.347 INFO:teuthology.orchestra.run.smithi087.stdout: mon host = 172.21.15.87 2023-12-24T17:57:14.347 INFO:teuthology.orchestra.run.smithi087.stdout: mon client directed command retry = 5 2023-12-24T17:57:14.347 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on pool no app = False 2023-12-24T17:57:14.347 INFO:teuthology.orchestra.run.smithi087.stdout: ms inject socket failures = 5000 2023-12-24T17:57:14.347 INFO:teuthology.orchestra.run.smithi087.stdout: fsid = 0f2278f5-9601-4df5-8109-682fbac125ff 2023-12-24T17:57:14.347 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:14.347 INFO:teuthology.orchestra.run.smithi087.stdout:[osd] 2023-12-24T17:57:14.347 INFO:teuthology.orchestra.run.smithi087.stdout: osd journal size = 100 2023-12-24T17:57:14.347 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:14.347 INFO:teuthology.orchestra.run.smithi087.stdout: osd scrub load threshold = 5.0 2023-12-24T17:57:14.347 INFO:teuthology.orchestra.run.smithi087.stdout: osd scrub max interval = 600 2023-12-24T17:57:14.347 INFO:teuthology.orchestra.run.smithi087.stdout: osd mclock profile = high_recovery_ops 2023-12-24T17:57:14.348 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:14.348 INFO:teuthology.orchestra.run.smithi087.stdout: osd recover clone overlap = true 2023-12-24T17:57:14.348 INFO:teuthology.orchestra.run.smithi087.stdout: osd recovery max chunk = 1048576 2023-12-24T17:57:14.348 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:14.348 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug shutdown = true 2023-12-24T17:57:14.348 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug op order = true 2023-12-24T17:57:14.348 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug verify stray on activate = true 2023-12-24T17:57:14.348 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:14.348 INFO:teuthology.orchestra.run.smithi087.stdout: osd open classes on start = true 2023-12-24T17:57:14.348 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug pg log writeout = true 2023-12-24T17:57:14.348 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:14.348 INFO:teuthology.orchestra.run.smithi087.stdout: osd deep scrub update digest min age = 30 2023-12-24T17:57:14.348 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:14.348 INFO:teuthology.orchestra.run.smithi087.stdout: osd map max advance = 10 2023-12-24T17:57:14.348 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:14.349 INFO:teuthology.orchestra.run.smithi087.stdout: journal zero on create = true 2023-12-24T17:57:14.349 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:14.349 INFO:teuthology.orchestra.run.smithi087.stdout: filestore ondisk finisher threads = 3 2023-12-24T17:57:14.349 INFO:teuthology.orchestra.run.smithi087.stdout: filestore apply finisher threads = 3 2023-12-24T17:57:14.349 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:14.349 INFO:teuthology.orchestra.run.smithi087.stdout: bdev debug aio = true 2023-12-24T17:57:14.349 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug misdirected ops = true 2023-12-24T17:57:14.349 INFO:teuthology.orchestra.run.smithi087.stdout: bdev async discard = True 2023-12-24T17:57:14.349 INFO:teuthology.orchestra.run.smithi087.stdout: bdev enable discard = True 2023-12-24T17:57:14.349 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore allocator = stupid 2023-12-24T17:57:14.349 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore block size = 96636764160 2023-12-24T17:57:14.349 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore fsck on mount = True 2023-12-24T17:57:14.349 INFO:teuthology.orchestra.run.smithi087.stdout: debug bluefs = 1/20 2023-12-24T17:57:14.349 INFO:teuthology.orchestra.run.smithi087.stdout: debug bluestore = 1/20 2023-12-24T17:57:14.349 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2023-12-24T17:57:14.350 INFO:teuthology.orchestra.run.smithi087.stdout: debug osd = 20 2023-12-24T17:57:14.350 INFO:teuthology.orchestra.run.smithi087.stdout: debug rocksdb = 4/10 2023-12-24T17:57:14.350 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd backfillfull_ratio = 0.85 2023-12-24T17:57:14.350 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd full ratio = 0.9 2023-12-24T17:57:14.350 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd nearfull ratio = 0.8 2023-12-24T17:57:14.350 INFO:teuthology.orchestra.run.smithi087.stdout: osd failsafe full ratio = 0.95 2023-12-24T17:57:14.350 INFO:teuthology.orchestra.run.smithi087.stdout: osd objectstore = bluestore 2023-12-24T17:57:14.350 INFO:teuthology.orchestra.run.smithi087.stdout: osd shutdown pgref assert = True 2023-12-24T17:57:14.350 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:14.350 INFO:teuthology.orchestra.run.smithi087.stdout:[mgr] 2023-12-24T17:57:14.350 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2023-12-24T17:57:14.350 INFO:teuthology.orchestra.run.smithi087.stdout: debug mgr = 20 2023-12-24T17:57:14.350 INFO:teuthology.orchestra.run.smithi087.stdout: debug mon = 20 2023-12-24T17:57:14.350 INFO:teuthology.orchestra.run.smithi087.stdout: debug auth = 20 2023-12-24T17:57:14.350 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min pgs per osd = 4 2023-12-24T17:57:14.351 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min bytes per osd = 10 2023-12-24T17:57:14.351 INFO:teuthology.orchestra.run.smithi087.stdout: mgr/telemetry/nag = false 2023-12-24T17:57:14.351 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:14.351 INFO:teuthology.orchestra.run.smithi087.stdout:[mon] 2023-12-24T17:57:14.351 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2023-12-24T17:57:14.351 INFO:teuthology.orchestra.run.smithi087.stdout: debug mon = 20 2023-12-24T17:57:14.351 INFO:teuthology.orchestra.run.smithi087.stdout: debug paxos = 20 2023-12-24T17:57:14.351 INFO:teuthology.orchestra.run.smithi087.stdout: debug auth = 20 2023-12-24T17:57:14.351 INFO:teuthology.orchestra.run.smithi087.stdout: mon data avail warn = 5 2023-12-24T17:57:14.351 INFO:teuthology.orchestra.run.smithi087.stdout: mon mgr mkfs grace = 240 2023-12-24T17:57:14.351 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min pgs per osd = 4 2023-12-24T17:57:14.351 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd reporter subtree level = osd 2023-12-24T17:57:14.351 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd prime pg temp = true 2023-12-24T17:57:14.352 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min bytes per osd = 10 2023-12-24T17:57:14.352 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:14.352 INFO:teuthology.orchestra.run.smithi087.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-24T17:57:14.352 INFO:teuthology.orchestra.run.smithi087.stdout: auth mon ticket ttl = 660 # 11m 2023-12-24T17:57:14.352 INFO:teuthology.orchestra.run.smithi087.stdout: auth service ticket ttl = 240 # 4m 2023-12-24T17:57:14.352 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:14.352 INFO:teuthology.orchestra.run.smithi087.stdout: # don't complain about insecure global_id in the test suite 2023-12-24T17:57:14.352 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-24T17:57:14.352 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-24T17:57:14.352 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:14.352 INFO:teuthology.orchestra.run.smithi087.stdout: # 1m isn't quite enough 2023-12-24T17:57:14.352 INFO:teuthology.orchestra.run.smithi087.stdout: mon_down_mkfs_grace = 2m 2023-12-24T17:57:14.352 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:14.352 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_filestore_osds = false 2023-12-24T17:57:14.353 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:14.353 INFO:teuthology.orchestra.run.smithi087.stdout:[client] 2023-12-24T17:57:14.353 INFO:teuthology.orchestra.run.smithi087.stdout: rgw cache enabled = true 2023-12-24T17:57:14.353 INFO:teuthology.orchestra.run.smithi087.stdout: rgw enable ops log = true 2023-12-24T17:57:14.353 INFO:teuthology.orchestra.run.smithi087.stdout: rgw enable usage log = true 2023-12-24T17:57:14.353 INFO:teuthology.orchestra.run.smithi087.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-24T17:57:14.353 INFO:teuthology.orchestra.run.smithi087.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-24T17:57:14.353 INFO:teuthology.orchestra.run.smithi087.stdout: rbd default format = 1 2023-12-24T17:57:14.353 INFO:teuthology.orchestra.run.smithi087.stdout:[mon.a] 2023-12-24T17:57:14.354 INFO:tasks.ceph:Creating admin key on mon.a... 2023-12-24T17:57:14.354 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-12-24T17:57:14.447 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-24T17:57:14.448 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-24T17:57:14.448 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-12-24T17:57:14.465 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-24T17:57:14.465 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-12-24T17:57:14.522 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi087.front.sepia.ceph.com 2023-12-24T17:57:14.523 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-24T17:57:14.523 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-24T17:57:14.523 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-24T17:57:14.597 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-24T17:57:14.598 DEBUG:teuthology.orchestra.run.smithi087:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-24T17:57:14.654 INFO:tasks.ceph:Setting up mon nodes... 2023-12-24T17:57:14.655 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-24T17:57:14.655 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-12-24T17:57:14.744 INFO:teuthology.orchestra.run.smithi087.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-12-24T17:57:14.746 INFO:tasks.ceph:Setting up mds nodes... 2023-12-24T17:57:14.746 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-24T17:57:14.747 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-12-24T17:57:14.786 INFO:teuthology.orchestra.run.smithi087.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-12-24T17:57:14.798 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-24T17:57:14.798 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi087.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-24T17:57:14.798 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-12-24T17:57:14.865 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-12-24T17:57:14.865 INFO:tasks.ceph:role: osd.0 2023-12-24T17:57:14.865 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi087.front.sepia.ceph.com 2023-12-24T17:57:14.865 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-24T17:57:14.939 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-24T17:57:14.939 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-24T17:57:14.940 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-24T17:57:14.940 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-24T17:57:14.940 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-24T17:57:14.940 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2023-12-24T17:57:14.940 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-24T17:57:14.940 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-24T17:57:14.940 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-24T17:57:14.940 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-24T17:57:15.369 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2023-12-24T17:57:15.370 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2023-12-24T17:57:15.370 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-12-24T17:57:15.543 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-12-24T17:57:15.576 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-12-24T17:57:15.644 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-12-24T17:57:15.644 INFO:tasks.ceph:role: osd.1 2023-12-24T17:57:15.644 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi087.front.sepia.ceph.com 2023-12-24T17:57:15.644 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-24T17:57:15.718 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-24T17:57:15.718 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-24T17:57:15.719 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-24T17:57:15.719 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-24T17:57:15.719 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-24T17:57:15.719 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2023-12-24T17:57:15.719 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-24T17:57:15.719 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-24T17:57:15.719 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-24T17:57:15.719 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-24T17:57:16.146 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2023-12-24T17:57:16.147 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2023-12-24T17:57:16.148 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-12-24T17:57:16.264 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-12-24T17:57:16.294 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-12-24T17:57:16.361 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-12-24T17:57:16.361 INFO:tasks.ceph:role: osd.2 2023-12-24T17:57:16.361 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi087.front.sepia.ceph.com 2023-12-24T17:57:16.362 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-24T17:57:16.436 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-24T17:57:16.436 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-24T17:57:16.436 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-24T17:57:16.436 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-24T17:57:16.436 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-24T17:57:16.436 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2023-12-24T17:57:16.436 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-24T17:57:16.436 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-24T17:57:16.436 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-24T17:57:16.436 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-24T17:57:16.894 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2023-12-24T17:57:16.896 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2023-12-24T17:57:16.896 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-12-24T17:57:17.045 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-12-24T17:57:17.079 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-24T17:57:17.171 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-24T17:57:17.173+0000 7f827ffbd540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-12-24T17:57:17.172 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-24T17:57:17.174+0000 7f827ffbd540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-12-24T17:57:17.172 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-24T17:57:17.174+0000 7f827ffbd540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-24T17:57:17.172 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-24T17:57:17.174+0000 7f827ffbd540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-24T17:57:17.172 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-24T17:57:17.174+0000 7f827ffbd540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-24T17:57:17.180 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-24T17:57:17.182+0000 7f827ffbd540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-12-24T17:57:17.708 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-24T17:57:17.710+0000 7f827ffbd540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-24T17:57:18.475 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-24T17:57:18.477+0000 7f827ffbd540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-24T17:57:19.235 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-24T17:57:19.238+0000 7f827ffbd540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-24T17:57:20.034 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-12-24T17:57:20.066 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-24T17:57:20.155 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-24T17:57:20.156+0000 7f7d84165540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-12-24T17:57:20.155 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-24T17:57:20.157+0000 7f7d84165540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-12-24T17:57:20.155 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-24T17:57:20.157+0000 7f7d84165540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-24T17:57:20.155 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-24T17:57:20.157+0000 7f7d84165540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-24T17:57:20.155 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-24T17:57:20.157+0000 7f7d84165540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-24T17:57:20.155 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-24T17:57:20.157+0000 7f7d84165540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-12-24T17:57:20.681 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-24T17:57:20.683+0000 7f7d84165540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-24T17:57:21.444 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-24T17:57:21.447+0000 7f7d84165540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-24T17:57:22.209 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-24T17:57:22.212+0000 7f7d84165540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-24T17:57:23.001 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-12-24T17:57:23.032 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-24T17:57:23.119 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-24T17:57:23.121+0000 7f41f2648540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-12-24T17:57:23.120 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-24T17:57:23.122+0000 7f41f2648540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-12-24T17:57:23.120 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-24T17:57:23.122+0000 7f41f2648540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-24T17:57:23.120 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-24T17:57:23.122+0000 7f41f2648540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-24T17:57:23.120 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-24T17:57:23.122+0000 7f41f2648540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-24T17:57:23.120 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-24T17:57:23.122+0000 7f41f2648540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-12-24T17:57:23.644 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-24T17:57:23.646+0000 7f41f2648540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-24T17:57:24.410 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-24T17:57:24.412+0000 7f41f2648540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-24T17:57:25.175 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-24T17:57:25.178+0000 7f41f2648540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-12-24T17:57:25.972 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-12-24T17:57:26.002 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-24T17:57:26.002 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-24T17:57:26.002 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-12-24T17:57:26.071 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-24T17:57:26.071 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-12-24T17:57:26.138 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-24T17:57:26.139 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-12-24T17:57:26.206 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-24T17:57:26.206 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-12-24T17:57:26.271 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-24T17:57:26.272 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-12-24T17:57:26.329 INFO:tasks.ceph:Adding keys to all mons... 2023-12-24T17:57:26.329 DEBUG:teuthology.orchestra.run.smithi087:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-24T17:57:26.395 INFO:teuthology.orchestra.run.smithi087.stdout:[mgr.x] 2023-12-24T17:57:26.395 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQD6cIhlr6OXLBAA0OxM4UN29+6pzu8UUu7UcQ== 2023-12-24T17:57:26.395 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.0] 2023-12-24T17:57:26.395 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQD9cIhl5NJwChAAVs+ziy7uCjkRQGH9ktOOaQ== 2023-12-24T17:57:26.395 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.1] 2023-12-24T17:57:26.395 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQAAcYhlzpNtCRAAL52V/Occva/wJVLAD+cnQA== 2023-12-24T17:57:26.395 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.2] 2023-12-24T17:57:26.395 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQADcYhl5iJYBxAAwu/lJq8WaCtE7XUDQL1k/Q== 2023-12-24T17:57:26.395 INFO:teuthology.orchestra.run.smithi087.stdout:[client.0] 2023-12-24T17:57:26.395 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQD6cIhltogcLxAAoPocanp4kSMTx47Ex40z+g== 2023-12-24T17:57:26.396 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-24T17:57:26.486 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-24T17:57:26.535 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-24T17:57:26.583 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-24T17:57:26.633 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-24T17:57:26.684 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-24T17:57:26.684 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-12-24T17:57:26.710 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-24T17:57:26.803 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-12-24T17:57:26.829 DEBUG:teuthology.orchestra.run.smithi087:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-24T17:57:26.888 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-12-24T17:57:26.888 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-24T17:57:26.888 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-12-24T17:57:26.930 INFO:tasks.ceph.mon.a:Started 2023-12-24T17:57:26.931 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-12-24T17:57:26.931 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-12-24T17:57:26.931 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-12-24T17:57:26.933 INFO:tasks.ceph.mgr.x:Started 2023-12-24T17:57:26.933 INFO:tasks.ceph:Setting crush tunables to default 2023-12-24T17:57:26.933 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd crush tunables default 2023-12-24T17:57:27.140 INFO:teuthology.orchestra.run.smithi087.stderr:adjusted tunables profile to default 2023-12-24T17:57:27.151 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-12-24T17:57:27.151 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-24T17:57:27.151 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-12-24T17:57:27.177 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-24T17:57:27.177 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-12-24T17:57:27.244 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-24T17:57:27.244 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-12-24T17:57:27.311 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd new 2c99abb1-7a79-494e-a998-e2942573c6ff 0 2023-12-24T17:57:27.327 INFO:tasks.ceph.mgr.x.smithi087.stderr:2023-12-24T17:57:27.330+0000 7ff5867d31c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-24T17:57:27.498 INFO:tasks.ceph.mgr.x.smithi087.stderr:2023-12-24T17:57:27.500+0000 7ff5867d31c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-24T17:57:27.563 INFO:teuthology.orchestra.run.smithi087.stdout:0 2023-12-24T17:57:27.575 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd new 2f9896f4-15d2-40c9-8bd7-05fe1e6d8b61 1 2023-12-24T17:57:27.797 INFO:teuthology.orchestra.run.smithi087.stdout:1 2023-12-24T17:57:27.809 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd new 0403cd42-78bb-43a2-9809-62b634e0e5df 2 2023-12-24T17:57:27.862 INFO:tasks.ceph.mgr.x.smithi087.stderr:2023-12-24T17:57:27.864+0000 7ff5867d31c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-24T17:57:28.017 INFO:teuthology.orchestra.run.smithi087.stdout:2 2023-12-24T17:57:28.029 INFO:tasks.ceph.osd.0:Restarting daemon 2023-12-24T17:57:28.029 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-12-24T17:57:28.031 INFO:tasks.ceph.osd.0:Started 2023-12-24T17:57:28.031 INFO:tasks.ceph.osd.1:Restarting daemon 2023-12-24T17:57:28.031 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-12-24T17:57:28.033 INFO:tasks.ceph.osd.1:Started 2023-12-24T17:57:28.034 INFO:tasks.ceph.osd.2:Restarting daemon 2023-12-24T17:57:28.034 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-12-24T17:57:28.036 INFO:tasks.ceph.osd.2:Started 2023-12-24T17:57:28.036 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-24T17:57:28.242 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:28.242 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":5,"fsid":"0f2278f5-9601-4df5-8109-682fbac125ff","created":"2023-12-24T17:57:27.057964+0000","modified":"2023-12-24T17:57:28.019393+0000","last_up_change":"0.000000","last_in_change":"2023-12-24T17:57:28.019393+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"2c99abb1-7a79-494e-a998-e2942573c6ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f9896f4-15d2-40c9-8bd7-05fe1e6d8b61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0403cd42-78bb-43a2-9809-62b634e0e5df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-24T17:57:28.254 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-12-24T17:57:28.255 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-24T17:57:28.388 INFO:tasks.ceph.mgr.x.smithi087.stderr:2023-12-24T17:57:28.391+0000 7ff5867d31c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-24T17:57:28.527 INFO:tasks.ceph.mgr.x.smithi087.stderr:2023-12-24T17:57:28.529+0000 7ff5867d31c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-24T17:57:28.556 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-24T17:57:28.644 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T17:57:28.647+0000 7f622edab540 -1 Falling back to public interface 2023-12-24T17:57:28.665 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T17:57:28.668+0000 7f7f7c7cd540 -1 Falling back to public interface 2023-12-24T17:57:28.669 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T17:57:28.671+0000 7f6daf016540 -1 Falling back to public interface 2023-12-24T17:57:28.745 INFO:teuthology.misc.health.smithi087.stdout: 2023-12-24T17:57:28.745 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":5,"fsid":"0f2278f5-9601-4df5-8109-682fbac125ff","created":"2023-12-24T17:57:27.057964+0000","modified":"2023-12-24T17:57:28.019393+0000","last_up_change":"0.000000","last_in_change":"2023-12-24T17:57:28.019393+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"2c99abb1-7a79-494e-a998-e2942573c6ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f9896f4-15d2-40c9-8bd7-05fe1e6d8b61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0403cd42-78bb-43a2-9809-62b634e0e5df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-24T17:57:28.756 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-24T17:57:28.889 INFO:tasks.ceph.mgr.x.smithi087.stderr:2023-12-24T17:57:28.891+0000 7ff5867d31c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-24T17:57:29.607 INFO:tasks.ceph.mgr.x.smithi087.stderr:2023-12-24T17:57:29.610+0000 7ff5867d31c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-24T17:57:29.678 INFO:tasks.ceph.mgr.x.smithi087.stderr:2023-12-24T17:57:29.680+0000 7ff5867d31c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-24T17:57:29.811 INFO:tasks.ceph.mgr.x.smithi087.stderr:2023-12-24T17:57:29.814+0000 7ff5867d31c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-24T17:57:29.878 INFO:tasks.ceph.mgr.x.smithi087.stderr:2023-12-24T17:57:29.880+0000 7ff5867d31c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-24T17:57:30.055 INFO:tasks.ceph.mgr.x.smithi087.stderr:2023-12-24T17:57:30.058+0000 7ff5867d31c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-24T17:57:30.131 INFO:tasks.ceph.mgr.x.smithi087.stderr:2023-12-24T17:57:30.134+0000 7ff5867d31c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-24T17:57:30.206 INFO:tasks.ceph.mgr.x.smithi087.stderr:2023-12-24T17:57:30.209+0000 7ff5867d31c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-24T17:57:30.299 INFO:tasks.ceph.mgr.x.smithi087.stderr:2023-12-24T17:57:30.301+0000 7ff5867d31c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-24T17:57:30.476 INFO:tasks.ceph.mgr.x.smithi087.stderr:2023-12-24T17:57:30.478+0000 7ff5867d31c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-24T17:57:30.555 INFO:tasks.ceph.mgr.x.smithi087.stderr:2023-12-24T17:57:30.557+0000 7ff5867d31c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-24T17:57:30.835 INFO:tasks.ceph.mgr.x.smithi087.stderr:2023-12-24T17:57:30.838+0000 7ff5867d31c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-24T17:57:30.904 INFO:tasks.ceph.mgr.x.smithi087.stderr:2023-12-24T17:57:30.907+0000 7ff5867d31c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-24T17:57:31.428 INFO:tasks.ceph.mgr.x.smithi087.stderr:2023-12-24T17:57:31.431+0000 7ff5867d31c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-24T17:57:31.761 INFO:tasks.ceph.mgr.x.smithi087.stderr:2023-12-24T17:57:31.764+0000 7ff5867d31c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-24T17:57:31.907 INFO:tasks.ceph.mgr.x.smithi087.stderr:2023-12-24T17:57:31.909+0000 7ff5867d31c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-24T17:57:32.165 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T17:57:32.168+0000 7f6daf016540 -1 osd.0 0 log_to_monitors true 2023-12-24T17:57:32.171 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T17:57:32.174+0000 7f7f7c7cd540 -1 osd.1 0 log_to_monitors true 2023-12-24T17:57:32.173 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T17:57:32.176+0000 7f622edab540 -1 osd.2 0 log_to_monitors true 2023-12-24T17:57:32.271 INFO:tasks.ceph.mgr.x.smithi087.stderr:2023-12-24T17:57:32.273+0000 7ff5867d31c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-24T17:57:32.413 INFO:tasks.ceph.mgr.x.smithi087.stderr:2023-12-24T17:57:32.415+0000 7ff5867d31c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-24T17:57:32.788 INFO:tasks.ceph.mgr.x.smithi087.stderr:2023-12-24T17:57:32.563+0000 7ff5867d31c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-24T17:57:32.788 INFO:tasks.ceph.mgr.x.smithi087.stderr:2023-12-24T17:57:32.666+0000 7ff5867d31c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-24T17:57:35.059 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-24T17:57:35.334 INFO:teuthology.misc.health.smithi087.stdout: 2023-12-24T17:57:35.335 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":7,"fsid":"0f2278f5-9601-4df5-8109-682fbac125ff","created":"2023-12-24T17:57:27.057964+0000","modified":"2023-12-24T17:57:34.020364+0000","last_up_change":"0.000000","last_in_change":"2023-12-24T17:57:28.019393+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"2c99abb1-7a79-494e-a998-e2942573c6ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f9896f4-15d2-40c9-8bd7-05fe1e6d8b61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0403cd42-78bb-43a2-9809-62b634e0e5df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-24T17:57:35.350 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-24T17:57:35.710 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T17:57:35.712+0000 7f6da5c1b700 -1 osd.0 0 waiting for initial osdmap 2023-12-24T17:57:35.718 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T17:57:35.720+0000 7f6d9ea12700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-24T17:57:35.736 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T17:57:35.738+0000 7f62259b0700 -1 osd.2 0 waiting for initial osdmap 2023-12-24T17:57:35.742 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T17:57:35.745+0000 7f621e7a7700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-24T17:57:35.830 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T17:57:35.833+0000 7f7f733d2700 -1 osd.1 0 waiting for initial osdmap 2023-12-24T17:57:35.836 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T17:57:35.838+0000 7f7f6c1c9700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-24T17:57:35.920 INFO:tasks.ceph.mgr.x.smithi087.stderr:2023-12-24T17:57:35.922+0000 7ff54aeb5700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-12-24T17:57:41.656 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-24T17:57:41.968 INFO:teuthology.misc.health.smithi087.stdout: 2023-12-24T17:57:41.968 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":11,"fsid":"0f2278f5-9601-4df5-8109-682fbac125ff","created":"2023-12-24T17:57:27.057964+0000","modified":"2023-12-24T17:57:40.084860+0000","last_up_change":"2023-12-24T17:57:36.716834+0000","last_in_change":"2023-12-24T17:57:28.019393+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-24T17:57:36.963424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"2c99abb1-7a79-494e-a998-e2942573c6ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":114598},{"type":"v1","addr":"172.21.15.87:6817","nonce":114598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":114598},{"type":"v1","addr":"172.21.15.87:6819","nonce":114598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":114598},{"type":"v1","addr":"172.21.15.87:6823","nonce":114598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":114598},{"type":"v1","addr":"172.21.15.87:6821","nonce":114598}]},"public_addr":"172.21.15.87:6817/114598","cluster_addr":"172.21.15.87:6819/114598","heartbeat_back_addr":"172.21.15.87:6823/114598","heartbeat_front_addr":"172.21.15.87:6821/114598","state":["exists","up"]},{"osd":1,"uuid":"2f9896f4-15d2-40c9-8bd7-05fe1e6d8b61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":114597},{"type":"v1","addr":"172.21.15.87:6809","nonce":114597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":114597},{"type":"v1","addr":"172.21.15.87:6811","nonce":114597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":114597},{"type":"v1","addr":"172.21.15.87:6815","nonce":114597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":114597},{"type":"v1","addr":"172.21.15.87:6813","nonce":114597}]},"public_addr":"172.21.15.87:6809/114597","cluster_addr":"172.21.15.87:6811/114597","heartbeat_back_addr":"172.21.15.87:6815/114597","heartbeat_front_addr":"172.21.15.87:6813/114597","state":["exists","up"]},{"osd":2,"uuid":"0403cd42-78bb-43a2-9809-62b634e0e5df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":114582},{"type":"v1","addr":"172.21.15.87:6801","nonce":114582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":114582},{"type":"v1","addr":"172.21.15.87:6803","nonce":114582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":114582},{"type":"v1","addr":"172.21.15.87:6807","nonce":114582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":114582},{"type":"v1","addr":"172.21.15.87:6805","nonce":114582}]},"public_addr":"172.21.15.87:6801/114582","cluster_addr":"172.21.15.87:6803/114582","heartbeat_back_addr":"172.21.15.87:6807/114582","heartbeat_front_addr":"172.21.15.87:6805/114582","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-24T17:57:33.148164+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-24T17:57:33.220152+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-24T17:57:33.135613+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-24T17:57:41.980 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-12-24T17:57:41.980 INFO:tasks.ceph:Creating RBD pool 2023-12-24T17:57:41.980 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd pool create rbd 8 2023-12-24T17:57:43.098 INFO:teuthology.orchestra.run.smithi087.stderr:pool 'rbd' created 2023-12-24T17:57:43.111 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-24T17:57:44.100 INFO:teuthology.orchestra.run.smithi087.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-24T17:57:44.111 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-12-24T17:57:44.112 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-12-24T17:57:44.112 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-12-24T17:57:44.112 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-12-24T17:57:44.112 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-24T17:57:44.473 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:44.485 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":5,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6824","nonce":114170},{"type":"v1","addr":"172.21.15.87:6825","nonce":114170}]},"active_addr":"172.21.15.87:6825/114170","active_change":"2023-12-24T17:57:34.692145+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.87:0","nonce":2107715108}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.87:0","nonce":511171699}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.87:0","nonce":475794130}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.87:0","nonce":2419276208}]}]} 2023-12-24T17:57:44.487 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-12-24T17:57:44.487 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-12-24T17:57:44.487 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-24T17:57:44.794 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:44.794 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":13,"fsid":"0f2278f5-9601-4df5-8109-682fbac125ff","created":"2023-12-24T17:57:27.057964+0000","modified":"2023-12-24T17:57:44.094041+0000","last_up_change":"2023-12-24T17:57:36.716834+0000","last_in_change":"2023-12-24T17:57:28.019393+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-24T17:57:36.963424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-24T17:57:42.305786+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2c99abb1-7a79-494e-a998-e2942573c6ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":114598},{"type":"v1","addr":"172.21.15.87:6817","nonce":114598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":114598},{"type":"v1","addr":"172.21.15.87:6819","nonce":114598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":114598},{"type":"v1","addr":"172.21.15.87:6823","nonce":114598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":114598},{"type":"v1","addr":"172.21.15.87:6821","nonce":114598}]},"public_addr":"172.21.15.87:6817/114598","cluster_addr":"172.21.15.87:6819/114598","heartbeat_back_addr":"172.21.15.87:6823/114598","heartbeat_front_addr":"172.21.15.87:6821/114598","state":["exists","up"]},{"osd":1,"uuid":"2f9896f4-15d2-40c9-8bd7-05fe1e6d8b61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":114597},{"type":"v1","addr":"172.21.15.87:6809","nonce":114597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":114597},{"type":"v1","addr":"172.21.15.87:6811","nonce":114597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":114597},{"type":"v1","addr":"172.21.15.87:6815","nonce":114597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":114597},{"type":"v1","addr":"172.21.15.87:6813","nonce":114597}]},"public_addr":"172.21.15.87:6809/114597","cluster_addr":"172.21.15.87:6811/114597","heartbeat_back_addr":"172.21.15.87:6815/114597","heartbeat_front_addr":"172.21.15.87:6813/114597","state":["exists","up"]},{"osd":2,"uuid":"0403cd42-78bb-43a2-9809-62b634e0e5df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":114582},{"type":"v1","addr":"172.21.15.87:6801","nonce":114582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":114582},{"type":"v1","addr":"172.21.15.87:6803","nonce":114582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":114582},{"type":"v1","addr":"172.21.15.87:6807","nonce":114582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":114582},{"type":"v1","addr":"172.21.15.87:6805","nonce":114582}]},"public_addr":"172.21.15.87:6801/114582","cluster_addr":"172.21.15.87:6803/114582","heartbeat_back_addr":"172.21.15.87:6807/114582","heartbeat_front_addr":"172.21.15.87:6805/114582","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-24T17:57:33.148164+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-24T17:57:33.220152+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-24T17:57:33.135613+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-24T17:57:44.807 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-12-24T17:57:44.807 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-24T17:57:45.115 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:45.115 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":14,"fsid":"0f2278f5-9601-4df5-8109-682fbac125ff","created":"2023-12-24T17:57:27.057964+0000","modified":"2023-12-24T17:57:45.098352+0000","last_up_change":"2023-12-24T17:57:36.716834+0000","last_in_change":"2023-12-24T17:57:28.019393+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-24T17:57:36.963424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-24T17:57:42.305786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2c99abb1-7a79-494e-a998-e2942573c6ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":114598},{"type":"v1","addr":"172.21.15.87:6817","nonce":114598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":114598},{"type":"v1","addr":"172.21.15.87:6819","nonce":114598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":114598},{"type":"v1","addr":"172.21.15.87:6823","nonce":114598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":114598},{"type":"v1","addr":"172.21.15.87:6821","nonce":114598}]},"public_addr":"172.21.15.87:6817/114598","cluster_addr":"172.21.15.87:6819/114598","heartbeat_back_addr":"172.21.15.87:6823/114598","heartbeat_front_addr":"172.21.15.87:6821/114598","state":["exists","up"]},{"osd":1,"uuid":"2f9896f4-15d2-40c9-8bd7-05fe1e6d8b61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":114597},{"type":"v1","addr":"172.21.15.87:6809","nonce":114597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":114597},{"type":"v1","addr":"172.21.15.87:6811","nonce":114597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":114597},{"type":"v1","addr":"172.21.15.87:6815","nonce":114597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":114597},{"type":"v1","addr":"172.21.15.87:6813","nonce":114597}]},"public_addr":"172.21.15.87:6809/114597","cluster_addr":"172.21.15.87:6811/114597","heartbeat_back_addr":"172.21.15.87:6815/114597","heartbeat_front_addr":"172.21.15.87:6813/114597","state":["exists","up"]},{"osd":2,"uuid":"0403cd42-78bb-43a2-9809-62b634e0e5df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":114582},{"type":"v1","addr":"172.21.15.87:6801","nonce":114582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":114582},{"type":"v1","addr":"172.21.15.87:6803","nonce":114582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":114582},{"type":"v1","addr":"172.21.15.87:6807","nonce":114582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":114582},{"type":"v1","addr":"172.21.15.87:6805","nonce":114582}]},"public_addr":"172.21.15.87:6801/114582","cluster_addr":"172.21.15.87:6803/114582","heartbeat_back_addr":"172.21.15.87:6807/114582","heartbeat_front_addr":"172.21.15.87:6805/114582","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-24T17:57:33.148164+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-24T17:57:33.220152+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-24T17:57:33.135613+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-24T17:57:45.129 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-12-24T17:57:45.129 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-12-24T17:57:45.130 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-12-24T17:57:45.294 INFO:teuthology.orchestra.run.smithi087.stdout:34359738372 2023-12-24T17:57:45.294 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-24T17:57:45.297 INFO:teuthology.orchestra.run.smithi087.stdout:34359738372 2023-12-24T17:57:45.297 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-24T17:57:45.304 INFO:teuthology.orchestra.run.smithi087.stdout:34359738371 2023-12-24T17:57:45.304 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-24T17:57:45.624 INFO:teuthology.orchestra.run.smithi087.stdout:34359738371 2023-12-24T17:57:45.637 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.2 2023-12-24T17:57:45.669 INFO:teuthology.orchestra.run.smithi087.stdout:34359738371 2023-12-24T17:57:45.674 INFO:teuthology.orchestra.run.smithi087.stdout:34359738370 2023-12-24T17:57:45.683 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.1 2023-12-24T17:57:45.688 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.0 2023-12-24T17:57:46.637 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-24T17:57:46.684 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-24T17:57:46.688 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-24T17:57:46.995 INFO:teuthology.orchestra.run.smithi087.stdout:34359738371 2023-12-24T17:57:47.009 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.2 2023-12-24T17:57:47.028 INFO:teuthology.orchestra.run.smithi087.stdout:34359738371 2023-12-24T17:57:47.042 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.1 2023-12-24T17:57:47.059 INFO:teuthology.orchestra.run.smithi087.stdout:34359738370 2023-12-24T17:57:47.072 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.0 2023-12-24T17:57:48.011 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-24T17:57:48.043 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-24T17:57:48.074 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-24T17:57:48.364 INFO:teuthology.orchestra.run.smithi087.stdout:34359738373 2023-12-24T17:57:48.378 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738373 for osd.2 2023-12-24T17:57:48.380 DEBUG:teuthology.parallel:result is None 2023-12-24T17:57:48.390 INFO:teuthology.orchestra.run.smithi087.stdout:34359738373 2023-12-24T17:57:48.404 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738373 for osd.1 2023-12-24T17:57:48.404 DEBUG:teuthology.parallel:result is None 2023-12-24T17:57:48.436 INFO:teuthology.orchestra.run.smithi087.stdout:34359738372 2023-12-24T17:57:48.449 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738372 for osd.0 2023-12-24T17:57:48.450 DEBUG:teuthology.parallel:result is None 2023-12-24T17:57:48.450 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-24T17:57:48.450 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T17:57:48.792 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:48.795 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T17:57:48.806 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2023-12-24T17:57:46.923551+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":893376,"kb_used_data":1512,"kb_used_omap":0,"kb_used_meta":891840,"kb_avail":282222144,"statfs":{"total":289910292480,"available":288995475456,"internally_reserved":0,"allocated":1548288,"data_stored":1281145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":913244160},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000623"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-24T17:57:44.103805+0000","last_change":"2023-12-24T17:57:44.101422+0000","last_active":"2023-12-24T17:57:44.103805+0000","last_peered":"2023-12-24T17:57:44.103805+0000","last_clean":"2023-12-24T17:57:44.103805+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T17:57:44.103805+0000","last_undegraded":"2023-12-24T17:57:44.103805+0000","last_fullsized":"2023-12-24T17:57:44.103805+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T20:39:55.726631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-24T17:57:44.103585+0000","last_change":"2023-12-24T17:57:44.101416+0000","last_active":"2023-12-24T17:57:44.103585+0000","last_peered":"2023-12-24T17:57:44.103585+0000","last_clean":"2023-12-24T17:57:44.103585+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T17:57:44.103585+0000","last_undegraded":"2023-12-24T17:57:44.103585+0000","last_fullsized":"2023-12-24T17:57:44.103585+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T01:55:17.265710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-24T17:57:44.103782+0000","last_change":"2023-12-24T17:57:44.101797+0000","last_active":"2023-12-24T17:57:44.103782+0000","last_peered":"2023-12-24T17:57:44.103782+0000","last_clean":"2023-12-24T17:57:44.103782+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T17:57:44.103782+0000","last_undegraded":"2023-12-24T17:57:44.103782+0000","last_fullsized":"2023-12-24T17:57:44.103782+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T00:30:36.804309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-24T17:57:44.103840+0000","last_change":"2023-12-24T17:57:44.102874+0000","last_active":"2023-12-24T17:57:44.103840+0000","last_peered":"2023-12-24T17:57:44.103840+0000","last_clean":"2023-12-24T17:57:44.103840+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T17:57:44.103840+0000","last_undegraded":"2023-12-24T17:57:44.103840+0000","last_fullsized":"2023-12-24T17:57:44.103840+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T19:39:13.450932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":139,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-24T17:57:44.096696+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T17:57:44.096696+0000","last_peered":"2023-12-24T17:57:44.096696+0000","last_clean":"2023-12-24T17:57:44.096696+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T17:57:44.096696+0000","last_undegraded":"2023-12-24T17:57:44.096696+0000","last_fullsized":"2023-12-24T17:57:44.096696+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T22:45:34.173764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-24T17:57:44.103500+0000","last_change":"2023-12-24T17:57:44.100967+0000","last_active":"2023-12-24T17:57:44.103500+0000","last_peered":"2023-12-24T17:57:44.103500+0000","last_clean":"2023-12-24T17:57:44.103500+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T17:57:44.103500+0000","last_undegraded":"2023-12-24T17:57:44.103500+0000","last_fullsized":"2023-12-24T17:57:44.103500+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T23:52:54.059450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-24T17:57:44.103854+0000","last_change":"2023-12-24T17:57:44.102897+0000","last_active":"2023-12-24T17:57:44.103854+0000","last_peered":"2023-12-24T17:57:44.103854+0000","last_clean":"2023-12-24T17:57:44.103854+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T17:57:44.103854+0000","last_undegraded":"2023-12-24T17:57:44.103854+0000","last_fullsized":"2023-12-24T17:57:44.103854+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T22:55:41.056162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-24T17:57:44.104186+0000","last_change":"2023-12-24T17:57:44.104186+0000","last_active":"2023-12-24T17:57:44.104186+0000","last_peered":"2023-12-24T17:57:44.104186+0000","last_clean":"2023-12-24T17:57:44.104186+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T17:57:44.104186+0000","last_undegraded":"2023-12-24T17:57:44.104186+0000","last_fullsized":"2023-12-24T17:57:44.104186+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T00:37:18.060893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-24T17:57:44.104355+0000","last_change":"2023-12-24T17:57:44.103195+0000","last_active":"2023-12-24T17:57:44.104355+0000","last_peered":"2023-12-24T17:57:44.104355+0000","last_clean":"2023-12-24T17:57:44.104355+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T17:57:44.104355+0000","last_undegraded":"2023-12-24T17:57:44.104355+0000","last_fullsized":"2023-12-24T17:57:44.104355+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T01:45:50.432394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297408,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074432,"statfs":{"total":96636764160,"available":96332218368,"internally_reserved":0,"allocated":122880,"data_stored":34625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297980,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073860,"statfs":{"total":96636764160,"available":96331632640,"internally_reserved":0,"allocated":708608,"data_stored":621527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25700000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297988,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073852,"statfs":{"total":96636764160,"available":96331624448,"internally_reserved":0,"allocated":716800,"data_stored":624993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T17:57:48.807 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T17:57:49.108 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:49.110 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T17:57:49.121 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2023-12-24T17:57:48.923909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":893376,"kb_used_data":1512,"kb_used_omap":0,"kb_used_meta":891840,"kb_avail":282222144,"statfs":{"total":289910292480,"available":288995475456,"internally_reserved":0,"allocated":1548288,"data_stored":1281145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":913244160},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000981"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-24T17:57:44.103805+0000","last_change":"2023-12-24T17:57:44.101422+0000","last_active":"2023-12-24T17:57:44.103805+0000","last_peered":"2023-12-24T17:57:44.103805+0000","last_clean":"2023-12-24T17:57:44.103805+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T17:57:44.103805+0000","last_undegraded":"2023-12-24T17:57:44.103805+0000","last_fullsized":"2023-12-24T17:57:44.103805+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T20:39:55.726631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-24T17:57:44.103585+0000","last_change":"2023-12-24T17:57:44.101416+0000","last_active":"2023-12-24T17:57:44.103585+0000","last_peered":"2023-12-24T17:57:44.103585+0000","last_clean":"2023-12-24T17:57:44.103585+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T17:57:44.103585+0000","last_undegraded":"2023-12-24T17:57:44.103585+0000","last_fullsized":"2023-12-24T17:57:44.103585+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T01:55:17.265710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-24T17:57:44.103782+0000","last_change":"2023-12-24T17:57:44.101797+0000","last_active":"2023-12-24T17:57:44.103782+0000","last_peered":"2023-12-24T17:57:44.103782+0000","last_clean":"2023-12-24T17:57:44.103782+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T17:57:44.103782+0000","last_undegraded":"2023-12-24T17:57:44.103782+0000","last_fullsized":"2023-12-24T17:57:44.103782+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T00:30:36.804309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-24T17:57:44.103840+0000","last_change":"2023-12-24T17:57:44.102874+0000","last_active":"2023-12-24T17:57:44.103840+0000","last_peered":"2023-12-24T17:57:44.103840+0000","last_clean":"2023-12-24T17:57:44.103840+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T17:57:44.103840+0000","last_undegraded":"2023-12-24T17:57:44.103840+0000","last_fullsized":"2023-12-24T17:57:44.103840+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T19:39:13.450932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":139,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-24T17:57:44.096696+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T17:57:44.096696+0000","last_peered":"2023-12-24T17:57:44.096696+0000","last_clean":"2023-12-24T17:57:44.096696+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T17:57:44.096696+0000","last_undegraded":"2023-12-24T17:57:44.096696+0000","last_fullsized":"2023-12-24T17:57:44.096696+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T22:45:34.173764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-24T17:57:44.103500+0000","last_change":"2023-12-24T17:57:44.100967+0000","last_active":"2023-12-24T17:57:44.103500+0000","last_peered":"2023-12-24T17:57:44.103500+0000","last_clean":"2023-12-24T17:57:44.103500+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T17:57:44.103500+0000","last_undegraded":"2023-12-24T17:57:44.103500+0000","last_fullsized":"2023-12-24T17:57:44.103500+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T23:52:54.059450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-24T17:57:44.103854+0000","last_change":"2023-12-24T17:57:44.102897+0000","last_active":"2023-12-24T17:57:44.103854+0000","last_peered":"2023-12-24T17:57:44.103854+0000","last_clean":"2023-12-24T17:57:44.103854+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T17:57:44.103854+0000","last_undegraded":"2023-12-24T17:57:44.103854+0000","last_fullsized":"2023-12-24T17:57:44.103854+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T22:55:41.056162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-24T17:57:44.104186+0000","last_change":"2023-12-24T17:57:44.104186+0000","last_active":"2023-12-24T17:57:44.104186+0000","last_peered":"2023-12-24T17:57:44.104186+0000","last_clean":"2023-12-24T17:57:44.104186+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T17:57:44.104186+0000","last_undegraded":"2023-12-24T17:57:44.104186+0000","last_fullsized":"2023-12-24T17:57:44.104186+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T00:37:18.060893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-24T17:57:44.104355+0000","last_change":"2023-12-24T17:57:44.103195+0000","last_active":"2023-12-24T17:57:44.104355+0000","last_peered":"2023-12-24T17:57:44.104355+0000","last_clean":"2023-12-24T17:57:44.104355+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T17:57:44.104355+0000","last_undegraded":"2023-12-24T17:57:44.104355+0000","last_fullsized":"2023-12-24T17:57:44.104355+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T01:45:50.432394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297408,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074432,"statfs":{"total":96636764160,"available":96332218368,"internally_reserved":0,"allocated":122880,"data_stored":34625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297980,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073860,"statfs":{"total":96636764160,"available":96331632640,"internally_reserved":0,"allocated":708608,"data_stored":621527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25700000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297988,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073852,"statfs":{"total":96636764160,"available":96331624448,"internally_reserved":0,"allocated":716800,"data_stored":624993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T17:57:49.122 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-24T17:57:49.122 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-12-24T17:57:49.122 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-12-24T17:57:49.122 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-12-24T17:57:49.472 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T17:57:49.472 INFO:teuthology.orchestra.run.smithi087.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-24T17:57:49.485 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-12-24T17:57:49.486 INFO:teuthology.run_tasks:Running task exec... 2023-12-24T17:57:49.496 INFO:teuthology.task.exec:Executing custom commands... 2023-12-24T17:57:49.496 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi087.front.sepia.ceph.com 2023-12-24T17:57:49.496 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool create cache 4' 2023-12-24T17:57:50.116 INFO:teuthology.orchestra.run.smithi087.stderr:pool 'cache' created 2023-12-24T17:57:50.129 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd tier add rbd cache' 2023-12-24T17:57:51.122 INFO:teuthology.orchestra.run.smithi087.stderr:pool 'cache' is now (or already was) a tier of 'rbd' 2023-12-24T17:57:51.136 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd tier cache-mode cache writeback' 2023-12-24T17:57:52.132 INFO:teuthology.orchestra.run.smithi087.stderr:set cache-mode for pool 'cache' to writeback 2023-12-24T17:57:52.146 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd tier set-overlay rbd cache' 2023-12-24T17:57:53.138 INFO:teuthology.orchestra.run.smithi087.stderr:overlay for 'rbd' is now (or already was) 'cache' 2023-12-24T17:57:53.152 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache hit_set_type bloom' 2023-12-24T17:57:54.146 INFO:teuthology.orchestra.run.smithi087.stderr:set pool 3 hit_set_type to bloom 2023-12-24T17:57:54.159 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache hit_set_count 8' 2023-12-24T17:57:55.148 INFO:teuthology.orchestra.run.smithi087.stderr:set pool 3 hit_set_count to 8 2023-12-24T17:57:55.162 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache hit_set_period 60' 2023-12-24T17:57:56.149 INFO:teuthology.orchestra.run.smithi087.stderr:set pool 3 hit_set_period to 60 2023-12-24T17:57:56.164 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache target_max_objects 250' 2023-12-24T17:57:57.154 INFO:teuthology.orchestra.run.smithi087.stderr:set pool 3 target_max_objects to 250 2023-12-24T17:57:57.168 INFO:teuthology.run_tasks:Running task workunit... 2023-12-24T17:57:57.181 INFO:tasks.workunit:Pulling workunits from ref 5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f 2023-12-24T17:57:57.181 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-12-24T17:57:57.181 DEBUG:teuthology.orchestra.run.smithi087:> stat -- /home/ubuntu/cephtest/mnt.0 2023-12-24T17:57:57.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-24T17:57:57.201 INFO:teuthology.orchestra.run.smithi087.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-12-24T17:57:57.201 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-12-24T17:57:57.260 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-12-24T17:57:57.260 DEBUG:teuthology.orchestra.run.smithi087:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-12-24T17:57:57.319 INFO:tasks.workunit:timeout=3h 2023-12-24T17:57:57.319 INFO:tasks.workunit:cleanup=True 2023-12-24T17:57:57.320 DEBUG:teuthology.orchestra.run.smithi087:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f 2023-12-24T17:57:57.379 INFO:tasks.workunit.client.0.smithi087.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-12-24T17:58:39.428 INFO:tasks.workunit.client.0.smithi087.stderr:Note: switching to '5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f'. 2023-12-24T17:58:39.428 INFO:tasks.workunit.client.0.smithi087.stderr: 2023-12-24T17:58:39.428 INFO:tasks.workunit.client.0.smithi087.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-24T17:58:39.428 INFO:tasks.workunit.client.0.smithi087.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-24T17:58:39.428 INFO:tasks.workunit.client.0.smithi087.stderr:state without impacting any branches by switching back to a branch. 2023-12-24T17:58:39.428 INFO:tasks.workunit.client.0.smithi087.stderr: 2023-12-24T17:58:39.428 INFO:tasks.workunit.client.0.smithi087.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-24T17:58:39.428 INFO:tasks.workunit.client.0.smithi087.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-24T17:58:39.428 INFO:tasks.workunit.client.0.smithi087.stderr: 2023-12-24T17:58:39.429 INFO:tasks.workunit.client.0.smithi087.stderr: git switch -c 2023-12-24T17:58:39.429 INFO:tasks.workunit.client.0.smithi087.stderr: 2023-12-24T17:58:39.429 INFO:tasks.workunit.client.0.smithi087.stderr:Or undo this operation with: 2023-12-24T17:58:39.429 INFO:tasks.workunit.client.0.smithi087.stderr: 2023-12-24T17:58:39.429 INFO:tasks.workunit.client.0.smithi087.stderr: git switch - 2023-12-24T17:58:39.429 INFO:tasks.workunit.client.0.smithi087.stderr: 2023-12-24T17:58:39.429 INFO:tasks.workunit.client.0.smithi087.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-24T17:58:39.429 INFO:tasks.workunit.client.0.smithi087.stderr: 2023-12-24T17:58:39.429 INFO:tasks.workunit.client.0.smithi087.stderr:HEAD is now at 5e7ce1294a2 Merge branch 'wip-63654-quincy' of https://github.com/idryomov/ceph into wip-yuri4-testing-2023-12-21-0917-quincy 2023-12-24T17:58:39.432 DEBUG:teuthology.orchestra.run.smithi087:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-12-24T17:58:39.500 INFO:tasks.workunit.client.0.smithi087.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-24T17:58:39.502 INFO:tasks.workunit.client.0.smithi087.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-24T17:58:39.503 INFO:tasks.workunit.client.0.smithi087.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-24T17:58:40.069 INFO:tasks.workunit.client.0.smithi087.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-24T17:58:40.128 INFO:tasks.workunit.client.0.smithi087.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-24T17:58:40.167 INFO:tasks.workunit.client.0.smithi087.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-24T17:58:40.168 INFO:tasks.workunit.client.0.smithi087.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-24T17:58:40.169 INFO:tasks.workunit.client.0.smithi087.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-24T17:58:40.205 INFO:tasks.workunit.client.0.smithi087.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-24T17:58:40.208 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-24T17:58:40.208 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-12-24T17:58:40.266 INFO:tasks.workunit:Running workunits matching rbd/cli_generic.sh on client.0... 2023-12-24T17:58:40.267 INFO:tasks.workunit:Running workunit rbd/cli_generic.sh... 2023-12-24T17:58:40.268 DEBUG:teuthology.orchestra.run.smithi087:workunit test rbd/cli_generic.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/cli_generic.sh 2023-12-24T17:58:40.328 INFO:tasks.workunit.client.0.smithi087.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/cli_generic.sh 2023-12-24T17:58:40.330 INFO:tasks.workunit.client.0.smithi087.stderr:+ . /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../standalone/ceph-helpers.sh 2023-12-24T17:58:40.330 INFO:tasks.workunit.client.0.smithi087.stderr:++ TIMEOUT=300 2023-12-24T17:58:40.330 INFO:tasks.workunit.client.0.smithi087.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2023-12-24T17:58:40.330 INFO:tasks.workunit.client.0.smithi087.stderr:++ MAX_TIMEOUT=15 2023-12-24T17:58:40.330 INFO:tasks.workunit.client.0.smithi087.stderr:++ PG_NUM=4 2023-12-24T17:58:40.330 INFO:tasks.workunit.client.0.smithi087.stderr:++ TMPDIR=/tmp 2023-12-24T17:58:40.330 INFO:tasks.workunit.client.0.smithi087.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2023-12-24T17:58:40.330 INFO:tasks.workunit.client.0.smithi087.stderr:++ TESTDIR=/home/ubuntu/cephtest 2023-12-24T17:58:40.331 INFO:tasks.workunit.client.0.smithi087.stderr:++ type xmlstarlet 2023-12-24T17:58:40.331 INFO:tasks.workunit.client.0.smithi087.stderr:++ XMLSTARLET=xmlstarlet 2023-12-24T17:58:40.331 INFO:tasks.workunit.client.0.smithi087.stderr:+++ uname 2023-12-24T17:58:40.331 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' Linux = FreeBSD ']' 2023-12-24T17:58:40.331 INFO:tasks.workunit.client.0.smithi087.stderr:++ SED=sed 2023-12-24T17:58:40.331 INFO:tasks.workunit.client.0.smithi087.stderr:++ AWK=awk 2023-12-24T17:58:40.332 INFO:tasks.workunit.client.0.smithi087.stderr:+++ stty -a 2023-12-24T17:58:40.332 INFO:tasks.workunit.client.0.smithi087.stderr:+++ head -1 2023-12-24T17:58:40.332 INFO:tasks.workunit.client.0.smithi087.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2023-12-24T17:58:40.332 INFO:tasks.workunit.client.0.smithi087.stderr:stty: 'standard input': Inappropriate ioctl for device 2023-12-24T17:58:40.332 INFO:tasks.workunit.client.0.smithi087.stderr:++ termwidth= 2023-12-24T17:58:40.332 INFO:tasks.workunit.client.0.smithi087.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2023-12-24T17:58:40.332 INFO:tasks.workunit.client.0.smithi087.stderr:++ DIFFCOLOPTS='-y ' 2023-12-24T17:58:40.332 INFO:tasks.workunit.client.0.smithi087.stderr:++ KERNCORE=kernel.core_pattern 2023-12-24T17:58:40.333 INFO:tasks.workunit.client.0.smithi087.stderr:++ EXTRA_OPTS= 2023-12-24T17:58:40.337 INFO:tasks.workunit.client.0.smithi087.stderr:++ test '' = TESTS 2023-12-24T17:58:40.337 INFO:tasks.workunit.client.0.smithi087.stderr:+ export RBD_FORCE_ALLOW_V1=1 2023-12-24T17:58:40.337 INFO:tasks.workunit.client.0.smithi087.stderr:+ RBD_FORCE_ALLOW_V1=1 2023-12-24T17:58:40.338 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T17:58:40.338 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T17:58:40.338 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -v '^0$' 2023-12-24T17:58:40.381 INFO:tasks.workunit.client.0.smithi087.stderr:+ IMGS='testimg1 testimg2 testimg3 testimg4 testimg5 testimg6 testimg-diff1 testimg-diff2 testimg-diff3 foo foo2 bar bar2 test1 test2 test3 test4 clone2' 2023-12-24T17:58:40.381 INFO:tasks.workunit.client.0.smithi087.stderr:+ tiered=0 2023-12-24T17:58:40.381 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph osd dump 2023-12-24T17:58:40.381 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^pool' 2023-12-24T17:58:40.381 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep ''\''rbd'\''' 2023-12-24T17:58:40.381 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep tier 2023-12-24T17:58:40.799 INFO:tasks.workunit.client.0.smithi087.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 18 lfor 18/18/18 flags hashpspool tiers 3 read_tier 3 write_tier 3 stripe_width 0 application rbd 2023-12-24T17:58:40.799 INFO:tasks.workunit.client.0.smithi087.stderr:+ tiered=1 2023-12-24T17:58:40.806 INFO:tasks.workunit.client.0.smithi087.stderr:+ test_pool_image_args 2023-12-24T17:58:40.806 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'testing pool and image args...' 2023-12-24T17:58:40.806 INFO:tasks.workunit.client.0.smithi087.stdout:testing pool and image args... 2023-12-24T17:58:40.807 INFO:tasks.workunit.client.0.smithi087.stderr:+ remove_images 2023-12-24T17:58:40.807 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:40.876 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:40.946 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:41.015 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:41.286 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:41.361 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:41.438 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:41.515 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:41.593 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:41.664 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:41.733 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:41.801 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:41.870 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:41.940 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:42.011 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:42.082 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:42.156 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:42.230 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:42.299 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph osd pool delete test test --yes-i-really-really-mean-it 2023-12-24T17:58:42.706 INFO:tasks.workunit.client.0.smithi087.stderr:pool 'test' does not exist 2023-12-24T17:58:42.715 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph osd pool create test 32 2023-12-24T17:58:43.325 INFO:tasks.workunit.client.0.smithi087.stderr:pool 'test' already exists 2023-12-24T17:58:43.335 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd pool init test 2023-12-24T17:58:46.216 INFO:tasks.workunit.client.0.smithi087.stderr:+ truncate -s 1 /tmp/empty /tmp/empty@snap 2023-12-24T17:58:46.230 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T17:58:46.230 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T17:58:46.230 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 0 2023-12-24T17:58:46.258 INFO:tasks.workunit.client.0.smithi087.stdout:0 2023-12-24T17:58:46.258 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create -s 1 test1 2023-12-24T17:58:46.287 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:58:46.289+0000 7f8746445540 -1 librbd: Forced V1 image creation. 2023-12-24T17:58:48.232 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T17:58:48.232 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -q test1 2023-12-24T17:58:48.261 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd import --image test2 /tmp/empty 2023-12-24T17:58:48.290 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:58:48.292+0000 7fc3dae69540 -1 librbd: Forced V1 image creation. 2023-12-24T17:58:48.300 INFO:tasks.workunit.client.0.smithi087.stderr: Importing image: 100% complete...done. 2023-12-24T17:58:48.300 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: --image is deprecated, use --dest 2023-12-24T17:58:48.302 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T17:58:48.303 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -q test2 2023-12-24T17:58:48.329 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd --dest test3 import /tmp/empty 2023-12-24T17:58:48.357 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:58:48.359+0000 7f5472547540 -1 librbd: Forced V1 image creation. 2023-12-24T17:58:48.368 INFO:tasks.workunit.client.0.smithi087.stderr: Importing image: 100% complete...done. 2023-12-24T17:58:48.371 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T17:58:48.371 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -q test3 2023-12-24T17:58:48.397 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd import /tmp/empty foo 2023-12-24T17:58:48.424 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:58:48.426+0000 7f4483b90540 -1 librbd: Forced V1 image creation. 2023-12-24T17:58:48.434 INFO:tasks.workunit.client.0.smithi087.stderr: Importing image: 100% complete...done. 2023-12-24T17:58:48.437 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T17:58:48.437 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -q foo 2023-12-24T17:58:48.463 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd import --dest test/empty@snap /tmp/empty 2023-12-24T17:58:48.485 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: destination snapshot name specified for a command that doesn't use it 2023-12-24T17:58:48.486 INFO:tasks.workunit.client.0.smithi087.stderr:+ true 2023-12-24T17:58:48.486 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd import /tmp/empty test/empty@snap 2023-12-24T17:58:48.506 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: destination snapshot name specified for a command that doesn't use it 2023-12-24T17:58:48.508 INFO:tasks.workunit.client.0.smithi087.stderr:+ true 2023-12-24T17:58:48.508 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd import --image test/empty@snap /tmp/empty 2023-12-24T17:58:48.528 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: destination snapshot name specified for a command that doesn't use it 2023-12-24T17:58:48.529 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: --image is deprecated, use --dest 2023-12-24T17:58:48.530 INFO:tasks.workunit.client.0.smithi087.stderr:+ true 2023-12-24T17:58:48.530 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd import /tmp/empty@snap 2023-12-24T17:58:48.549 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: destination snapshot name specified for a command that doesn't use it 2023-12-24T17:58:48.550 INFO:tasks.workunit.client.0.smithi087.stderr:+ true 2023-12-24T17:58:48.550 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls test 2023-12-24T17:58:48.550 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T17:58:48.550 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 0 2023-12-24T17:58:48.576 INFO:tasks.workunit.client.0.smithi087.stdout:0 2023-12-24T17:58:48.577 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd import /tmp/empty test/test1 2023-12-24T17:58:48.603 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:58:48.605+0000 7f20a8a37540 -1 librbd: Forced V1 image creation. 2023-12-24T17:58:50.244 INFO:tasks.workunit.client.0.smithi087.stderr: Importing image: 100% complete...done. 2023-12-24T17:58:50.247 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls test 2023-12-24T17:58:50.247 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -q test1 2023-12-24T17:58:50.278 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd -p test import /tmp/empty test2 2023-12-24T17:58:50.308 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:58:50.310+0000 7f7459bcf540 -1 librbd: Forced V1 image creation. 2023-12-24T17:58:50.317 INFO:tasks.workunit.client.0.smithi087.stderr: Importing image: 100% complete...done. 2023-12-24T17:58:50.318 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: -p [ --pool ] is deprecated, use --dest-pool 2023-12-24T17:58:50.320 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls test 2023-12-24T17:58:50.320 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -q test2 2023-12-24T17:58:50.348 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd --image test3 -p test import /tmp/empty 2023-12-24T17:58:50.378 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:58:50.381+0000 7f2fa7b21540 -1 librbd: Forced V1 image creation. 2023-12-24T17:58:50.389 INFO:tasks.workunit.client.0.smithi087.stderr: Importing image: 100% complete...done. 2023-12-24T17:58:50.389 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: --image is deprecated, use --dest 2023-12-24T17:58:50.389 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: -p [ --pool ] is deprecated, use --dest-pool 2023-12-24T17:58:50.391 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls test 2023-12-24T17:58:50.392 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -q test3 2023-12-24T17:58:50.419 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd --image test4 -p test import /tmp/empty 2023-12-24T17:58:50.448 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:58:50.451+0000 7f989c14b540 -1 librbd: Forced V1 image creation. 2023-12-24T17:58:50.463 INFO:tasks.workunit.client.0.smithi087.stderr: Importing image: 100% complete...done. 2023-12-24T17:58:50.464 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: --image is deprecated, use --dest 2023-12-24T17:58:50.464 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: -p [ --pool ] is deprecated, use --dest-pool 2023-12-24T17:58:50.466 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls test 2023-12-24T17:58:50.466 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -q test4 2023-12-24T17:58:50.494 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd --dest test5 -p test import /tmp/empty 2023-12-24T17:58:50.522 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:58:50.525+0000 7f06bccbe540 -1 librbd: Forced V1 image creation. 2023-12-24T17:58:50.538 INFO:tasks.workunit.client.0.smithi087.stderr: Importing image: 100% complete...done. 2023-12-24T17:58:50.538 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: -p [ --pool ] is deprecated, use --dest-pool 2023-12-24T17:58:50.541 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls test 2023-12-24T17:58:50.541 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -q test5 2023-12-24T17:58:50.568 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd --dest test6 --dest-pool test import /tmp/empty 2023-12-24T17:58:50.597 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:58:50.599+0000 7f898d217540 -1 librbd: Forced V1 image creation. 2023-12-24T17:58:50.614 INFO:tasks.workunit.client.0.smithi087.stderr: Importing image: 100% complete...done. 2023-12-24T17:58:50.616 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls test 2023-12-24T17:58:50.616 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -q test6 2023-12-24T17:58:50.644 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd --image test7 --dest-pool test import /tmp/empty 2023-12-24T17:58:50.673 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:58:50.675+0000 7f03e3abb540 -1 librbd: Forced V1 image creation. 2023-12-24T17:58:50.684 INFO:tasks.workunit.client.0.smithi087.stderr: Importing image: 100% complete...done. 2023-12-24T17:58:50.684 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: --image is deprecated, use --dest 2023-12-24T17:58:50.686 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls test 2023-12-24T17:58:50.687 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -q test7 2023-12-24T17:58:50.714 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd --image test/test8 import /tmp/empty 2023-12-24T17:58:50.742 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:58:50.744+0000 7f70ee2cc540 -1 librbd: Forced V1 image creation. 2023-12-24T17:58:50.751 INFO:tasks.workunit.client.0.smithi087.stderr: Importing image: 100% complete...done. 2023-12-24T17:58:50.752 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: --image is deprecated, use --dest 2023-12-24T17:58:50.754 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls test 2023-12-24T17:58:50.754 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -q test8 2023-12-24T17:58:50.782 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd --dest test/test9 import /tmp/empty 2023-12-24T17:58:50.808 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:58:50.811+0000 7f86f166b540 -1 librbd: Forced V1 image creation. 2023-12-24T17:58:50.817 INFO:tasks.workunit.client.0.smithi087.stderr: Importing image: 100% complete...done. 2023-12-24T17:58:50.819 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls test 2023-12-24T17:58:50.820 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -q test9 2023-12-24T17:58:50.847 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd import --pool test /tmp/empty 2023-12-24T17:58:51.076 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:58:51.078+0000 7f2de0bbb540 -1 librbd: Forced V1 image creation. 2023-12-24T17:58:51.087 INFO:tasks.workunit.client.0.smithi087.stderr: Importing image: 100% complete...done. 2023-12-24T17:58:51.087 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: -p [ --pool ] is deprecated, use --dest-pool 2023-12-24T17:58:51.090 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls test 2023-12-24T17:58:51.090 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -q empty 2023-12-24T17:58:51.118 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd copy test/test9 test10 2023-12-24T17:58:52.054 INFO:tasks.workunit.client.0.smithi087.stderr: Image copy: 100% complete... Image copy: 100% complete...done. 2023-12-24T17:58:52.058 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls test 2023-12-24T17:58:52.058 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -qv test10 2023-12-24T17:58:52.088 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T17:58:52.088 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -q test10 2023-12-24T17:58:52.115 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd copy test/test9 test/test10 2023-12-24T17:58:52.167 INFO:tasks.workunit.client.0.smithi087.stderr: Image copy: 100% complete... Image copy: 100% complete...done. 2023-12-24T17:58:52.170 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls test 2023-12-24T17:58:52.170 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -q test10 2023-12-24T17:58:52.198 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd copy --pool test test10 --dest-pool test test11 2023-12-24T17:58:52.253 INFO:tasks.workunit.client.0.smithi087.stderr: Image copy: 100% complete... Image copy: 100% complete...done. 2023-12-24T17:58:52.256 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls test 2023-12-24T17:58:52.256 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -q test11 2023-12-24T17:58:52.284 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd copy --dest-pool rbd --pool test test11 test12 2023-12-24T17:58:52.342 INFO:tasks.workunit.client.0.smithi087.stderr: Image copy: 100% complete... Image copy: 100% complete...done. 2023-12-24T17:58:52.345 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T17:58:52.345 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep test12 2023-12-24T17:58:52.372 INFO:tasks.workunit.client.0.smithi087.stdout:test12 2023-12-24T17:58:52.372 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls test 2023-12-24T17:58:52.372 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -qv test12 2023-12-24T17:58:52.399 INFO:tasks.workunit.client.0.smithi087.stderr:+ rm -f /tmp/empty /tmp/empty@snap 2023-12-24T17:58:52.400 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph osd pool delete test test --yes-i-really-really-mean-it 2023-12-24T17:58:53.361 INFO:tasks.workunit.client.0.smithi087.stderr:pool 'test' does not exist 2023-12-24T17:58:53.371 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in foo test1 test10 test12 test2 test3 2023-12-24T17:58:53.371 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm foo 2023-12-24T17:58:53.418 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:58:53.420 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in foo test1 test10 test12 test2 test3 2023-12-24T17:58:53.420 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test1 2023-12-24T17:58:53.456 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:58:53.458 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in foo test1 test10 test12 test2 test3 2023-12-24T17:58:53.458 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test10 2023-12-24T17:58:53.539 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:58:53.541 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in foo test1 test10 test12 test2 test3 2023-12-24T17:58:53.542 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test12 2023-12-24T17:58:53.625 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:58:53.628 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in foo test1 test10 test12 test2 test3 2023-12-24T17:58:53.628 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test2 2023-12-24T17:58:53.666 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:58:53.668 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in foo test1 test10 test12 test2 test3 2023-12-24T17:58:53.669 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test3 2023-12-24T17:58:53.703 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:58:53.705 INFO:tasks.workunit.client.0.smithi087.stderr:+ test_rename 2023-12-24T17:58:53.706 INFO:tasks.workunit.client.0.smithi087.stdout:testing rename... 2023-12-24T17:58:53.706 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'testing rename...' 2023-12-24T17:58:53.706 INFO:tasks.workunit.client.0.smithi087.stderr:+ remove_images 2023-12-24T17:58:53.706 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:53.774 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:53.842 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:53.909 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:53.978 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:54.047 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:54.115 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:54.183 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:54.252 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:54.321 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:54.398 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:54.475 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:54.553 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:54.628 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:54.702 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:54.776 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:54.845 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:54.913 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:58:54.983 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 1 -s 1 foo 2023-12-24T17:58:55.006 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: image format 1 is deprecated 2023-12-24T17:58:55.012 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:58:55.014+0000 7f8011c44540 -1 librbd: Forced V1 image creation. 2023-12-24T17:58:55.018 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 -s 1 bar 2023-12-24T17:58:55.059 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rename foo foo2 2023-12-24T17:58:55.101 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rename foo2 bar 2023-12-24T17:58:55.101 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep exists 2023-12-24T17:58:55.135 INFO:tasks.workunit.client.0.smithi087.stdout:2023-12-24T17:58:55.134+0000 7fa325c1d540 -1 librbd::Operations: rbd image bar already exists 2023-12-24T17:58:55.135 INFO:tasks.workunit.client.0.smithi087.stdout:rbd: rename error: (17) File exists 2023-12-24T17:58:55.135 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rename bar bar2 2023-12-24T17:58:55.194 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rename bar2 foo2 2023-12-24T17:58:55.194 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep exists 2023-12-24T17:58:55.232 INFO:tasks.workunit.client.0.smithi087.stdout:2023-12-24T17:58:55.228+0000 7fbfdba11540 -1 librbd::Operations: rbd image foo2 already exists 2023-12-24T17:58:55.232 INFO:tasks.workunit.client.0.smithi087.stdout:rbd: rename error: (17) File exists 2023-12-24T17:58:55.232 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph osd pool create rbd2 8 2023-12-24T17:58:56.369 INFO:tasks.workunit.client.0.smithi087.stderr:pool 'rbd2' already exists 2023-12-24T17:58:56.378 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd pool init rbd2 2023-12-24T17:58:59.044 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create -p rbd2 -s 1 foo 2023-12-24T17:58:59.077 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:58:59.079+0000 7f7d04d14540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:01.050 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rename rbd2/foo rbd2/bar 2023-12-24T17:59:01.097 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd -p rbd2 ls 2023-12-24T17:59:01.097 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep bar 2023-12-24T17:59:01.126 INFO:tasks.workunit.client.0.smithi087.stdout:bar 2023-12-24T17:59:01.126 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rename rbd2/bar foo 2023-12-24T17:59:01.169 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rename --pool rbd2 foo bar 2023-12-24T17:59:01.214 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rename rbd2/bar --dest-pool rbd foo 2023-12-24T17:59:01.237 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: mv/rename across pools not supported 2023-12-24T17:59:01.237 INFO:tasks.workunit.client.0.smithi087.stderr:source pool: rbd2 dest pool: rbd 2023-12-24T17:59:01.238 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rename --pool rbd2 bar --dest-pool rbd2 foo 2023-12-24T17:59:01.289 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd -p rbd2 ls 2023-12-24T17:59:01.289 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep foo 2023-12-24T17:59:01.317 INFO:tasks.workunit.client.0.smithi087.stdout:foo 2023-12-24T17:59:01.317 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph osd pool rm rbd2 rbd2 --yes-i-really-really-mean-it 2023-12-24T17:59:02.165 INFO:tasks.workunit.client.0.smithi087.stderr:pool 'rbd2' does not exist 2023-12-24T17:59:02.174 INFO:tasks.workunit.client.0.smithi087.stderr:+ remove_images 2023-12-24T17:59:02.174 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:02.254 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:02.322 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:02.391 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:02.460 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:02.529 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:02.597 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:02.666 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:02.734 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:02.802 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:02.870 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:02.949 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:03.023 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:03.145 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:03.213 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:03.282 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:03.351 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:03.426 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:03.502 INFO:tasks.workunit.client.0.smithi087.stderr:+ test_ls 2023-12-24T17:59:03.502 INFO:tasks.workunit.client.0.smithi087.stdout:testing ls... 2023-12-24T17:59:03.502 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'testing ls...' 2023-12-24T17:59:03.502 INFO:tasks.workunit.client.0.smithi087.stderr:+ remove_images 2023-12-24T17:59:03.502 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:03.577 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:03.648 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:03.716 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:03.785 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:03.855 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:03.923 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:03.991 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:04.065 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:04.143 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:04.221 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:04.295 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:04.372 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:04.448 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:04.516 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:04.583 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:04.651 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:04.719 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:04.988 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 1 -s 1 test1 2023-12-24T17:59:05.011 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: image format 1 is deprecated 2023-12-24T17:59:05.017 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:05.019+0000 7f7bf8372540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:05.022 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 1 -s 1 test2 2023-12-24T17:59:05.045 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: image format 1 is deprecated 2023-12-24T17:59:05.051 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:05.053+0000 7fe9e5be3540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:05.056 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T17:59:05.056 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep test1 2023-12-24T17:59:05.083 INFO:tasks.workunit.client.0.smithi087.stdout:test1 2023-12-24T17:59:05.083 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T17:59:05.083 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep test2 2023-12-24T17:59:05.110 INFO:tasks.workunit.client.0.smithi087.stdout:test2 2023-12-24T17:59:05.111 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T17:59:05.111 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T17:59:05.111 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 2 2023-12-24T17:59:05.137 INFO:tasks.workunit.client.0.smithi087.stdout:2 2023-12-24T17:59:05.137 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls -l 2023-12-24T17:59:05.137 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'test1.*1 MiB.*1' 2023-12-24T17:59:05.170 INFO:tasks.workunit.client.0.smithi087.stdout:test1 1 MiB 1 2023-12-24T17:59:05.170 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls -l 2023-12-24T17:59:05.171 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'test2.*1 MiB.*1' 2023-12-24T17:59:05.203 INFO:tasks.workunit.client.0.smithi087.stdout:test2 1 MiB 1 2023-12-24T17:59:05.204 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test1 2023-12-24T17:59:05.238 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:05.241 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test2 2023-12-24T17:59:05.276 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:05.278 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 -s 1 test1 2023-12-24T17:59:05.319 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 -s 1 test2 2023-12-24T17:59:05.368 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T17:59:05.368 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep test1 2023-12-24T17:59:05.394 INFO:tasks.workunit.client.0.smithi087.stdout:test1 2023-12-24T17:59:05.394 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T17:59:05.394 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep test2 2023-12-24T17:59:05.420 INFO:tasks.workunit.client.0.smithi087.stdout:test2 2023-12-24T17:59:05.421 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T17:59:05.421 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T17:59:05.421 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 2 2023-12-24T17:59:05.447 INFO:tasks.workunit.client.0.smithi087.stdout:2 2023-12-24T17:59:05.447 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls -l 2023-12-24T17:59:05.447 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'test1.*1 MiB.*2' 2023-12-24T17:59:05.483 INFO:tasks.workunit.client.0.smithi087.stdout:test1 1 MiB 2 2023-12-24T17:59:05.483 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls -l 2023-12-24T17:59:05.483 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'test2.*1 MiB.*2' 2023-12-24T17:59:05.518 INFO:tasks.workunit.client.0.smithi087.stdout:test2 1 MiB 2 2023-12-24T17:59:05.518 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test1 2023-12-24T17:59:05.619 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:05.622+0000 7f6bfbd63700 0 -- 172.21.15.87:0/464712952 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f6bd8008740 msgr2=0x7f6bd800abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:05.620 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:05.622 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test2 2023-12-24T17:59:05.697 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:05.700+0000 7f9a77cb3700 0 -- 172.21.15.87:0/39806918 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f9a54008740 msgr2=0x7f9a5400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:05.698 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:05.700 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 -s 1 test1 2023-12-24T17:59:05.742 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 1 -s 1 test2 2023-12-24T17:59:05.766 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: image format 1 is deprecated 2023-12-24T17:59:05.771 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:05.774+0000 7fef999ee540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:05.776 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T17:59:05.776 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep test1 2023-12-24T17:59:05.802 INFO:tasks.workunit.client.0.smithi087.stdout:test1 2023-12-24T17:59:05.803 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T17:59:05.803 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep test2 2023-12-24T17:59:05.829 INFO:tasks.workunit.client.0.smithi087.stdout:test2 2023-12-24T17:59:05.830 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T17:59:05.830 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T17:59:05.830 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 2 2023-12-24T17:59:05.856 INFO:tasks.workunit.client.0.smithi087.stdout:2 2023-12-24T17:59:05.856 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls -l 2023-12-24T17:59:05.856 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'test1.*1 MiB.*2' 2023-12-24T17:59:05.891 INFO:tasks.workunit.client.0.smithi087.stdout:test1 1 MiB 2 2023-12-24T17:59:05.892 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls -l 2023-12-24T17:59:05.892 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'test2.*1 MiB.*1' 2023-12-24T17:59:05.927 INFO:tasks.workunit.client.0.smithi087.stdout:test2 1 MiB 1 2023-12-24T17:59:05.928 INFO:tasks.workunit.client.0.smithi087.stderr:+ remove_images 2023-12-24T17:59:05.928 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:05.995 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:06.063 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:06.130 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:06.199 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:06.273 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:06.343 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:06.413 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:06.480 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:06.548 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:06.617 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:06.684 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:06.752 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:06.830 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:06.965 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:07.046 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:07.117 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:07.185 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:07.256 INFO:tasks.workunit.client.0.smithi087.stderr:++ seq -w 00 99 2023-12-24T17:59:07.269 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:07.269 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.00 -s 1 2023-12-24T17:59:07.297 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:07.300+0000 7ff9be1a3540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:07.306 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:07.306 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.01 -s 1 2023-12-24T17:59:07.333 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:07.336+0000 7f37071de540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:07.342 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:07.342 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.02 -s 1 2023-12-24T17:59:07.370 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:07.372+0000 7fe96257f540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:07.376 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:07.376 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.03 -s 1 2023-12-24T17:59:07.404 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:07.406+0000 7f93d7bc7540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:07.411 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:07.411 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.04 -s 1 2023-12-24T17:59:07.439 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:07.441+0000 7f4bd2aa6540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:07.446 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:07.446 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.05 -s 1 2023-12-24T17:59:07.474 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:07.476+0000 7fb4ee07a540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:07.485 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:07.485 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.06 -s 1 2023-12-24T17:59:07.512 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:07.515+0000 7ff955a3a540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:07.522 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:07.523 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.07 -s 1 2023-12-24T17:59:07.550 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:07.552+0000 7f7081300540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:07.558 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:07.558 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.08 -s 1 2023-12-24T17:59:07.586 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:07.588+0000 7f95c8e94540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:07.592 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:07.592 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.09 -s 1 2023-12-24T17:59:07.620 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:07.622+0000 7f45a13cc540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:07.626 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:07.627 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.10 -s 1 2023-12-24T17:59:07.654 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:07.657+0000 7f7ce6762540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:07.663 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:07.663 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.11 -s 1 2023-12-24T17:59:07.691 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:07.694+0000 7f70a1470540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:07.702 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:07.702 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.12 -s 1 2023-12-24T17:59:07.729 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:07.732+0000 7f0b3990a540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:07.740 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:07.740 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.13 -s 1 2023-12-24T17:59:07.768 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:07.770+0000 7fe447de6540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:07.778 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:07.778 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.14 -s 1 2023-12-24T17:59:07.806 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:07.809+0000 7fe8bec71540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:07.811 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:07.812 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.15 -s 1 2023-12-24T17:59:07.840 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:07.842+0000 7f0c62c3b540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:07.845 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:07.845 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.16 -s 1 2023-12-24T17:59:07.872 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:07.875+0000 7fc5c6375540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:07.878 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:07.878 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.17 -s 1 2023-12-24T17:59:07.906 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:07.908+0000 7f8ffc7a9540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:07.911 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:07.911 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.18 -s 1 2023-12-24T17:59:07.939 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:07.941+0000 7fa467aa0540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:07.944 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:07.944 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.19 -s 1 2023-12-24T17:59:07.972 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:07.974+0000 7f224796e540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:07.977 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:07.977 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.20 -s 1 2023-12-24T17:59:08.005 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:08.007+0000 7fd5bab36540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:08.009 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:08.009 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.21 -s 1 2023-12-24T17:59:08.041 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:08.041+0000 7f4955d0c540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:08.044 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:08.044 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.22 -s 1 2023-12-24T17:59:08.071 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:08.074+0000 7f2c2d66b540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:08.080 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:08.080 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.23 -s 1 2023-12-24T17:59:08.108 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:08.111+0000 7fc0e7c26540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:08.113 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:08.114 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.24 -s 1 2023-12-24T17:59:08.142 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:08.144+0000 7efceeed3540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:08.149 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:08.149 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.25 -s 1 2023-12-24T17:59:08.176 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:08.179+0000 7fd0bfc93540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:08.184 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:08.184 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.26 -s 1 2023-12-24T17:59:08.212 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:08.214+0000 7f61ed6d2540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:08.218 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:08.218 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.27 -s 1 2023-12-24T17:59:08.246 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:08.249+0000 7f504e537540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:08.254 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:08.255 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.28 -s 1 2023-12-24T17:59:08.283 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:08.285+0000 7f070eb89540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:08.287 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:08.288 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.29 -s 1 2023-12-24T17:59:08.315 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:08.318+0000 7febb055a540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:08.322 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:08.322 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.30 -s 1 2023-12-24T17:59:08.350 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:08.353+0000 7fab12ef5540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:08.358 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:08.358 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.31 -s 1 2023-12-24T17:59:08.386 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:08.389+0000 7f9e7900c540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:08.395 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:08.395 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.32 -s 1 2023-12-24T17:59:08.423 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:08.425+0000 7fcdec562540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:08.429 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:08.430 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.33 -s 1 2023-12-24T17:59:08.458 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:08.460+0000 7fe479847540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:08.464 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:08.464 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.34 -s 1 2023-12-24T17:59:08.492 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:08.494+0000 7f66fb0b3540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:08.501 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:08.501 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.35 -s 1 2023-12-24T17:59:08.530 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:08.533+0000 7f2166789540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:08.535 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:08.536 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.36 -s 1 2023-12-24T17:59:08.563 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:08.565+0000 7f7141e4e540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:08.568 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:08.568 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.37 -s 1 2023-12-24T17:59:08.597 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:08.599+0000 7f39168ae540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:08.608 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:08.608 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.38 -s 1 2023-12-24T17:59:08.635 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:08.638+0000 7f8e6c3e5540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:08.643 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:08.643 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.39 -s 1 2023-12-24T17:59:08.671 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:08.674+0000 7fe77fa93540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:08.677 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:08.678 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.40 -s 1 2023-12-24T17:59:08.705 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:08.708+0000 7f686e6ae540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:08.710 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:08.711 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.41 -s 1 2023-12-24T17:59:08.738 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:08.741+0000 7f77680a0540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:08.744 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:08.744 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.42 -s 1 2023-12-24T17:59:08.772 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:08.774+0000 7f9a1764b540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:08.777 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:08.777 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.43 -s 1 2023-12-24T17:59:08.804 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:08.807+0000 7f0519735540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:08.809 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:08.809 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.44 -s 1 2023-12-24T17:59:08.837 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:08.840+0000 7f5ba6c18540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:08.843 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:08.844 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.45 -s 1 2023-12-24T17:59:08.871 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:08.874+0000 7f251d396540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:08.876 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:08.877 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.46 -s 1 2023-12-24T17:59:08.904 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:08.907+0000 7fb08c05e540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:08.913 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:08.913 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.47 -s 1 2023-12-24T17:59:08.941 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:08.944+0000 7fb9ef5ad540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:08.952 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:08.953 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.48 -s 1 2023-12-24T17:59:08.981 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:08.984+0000 7f41bbae0540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:08.990 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:08.991 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.49 -s 1 2023-12-24T17:59:09.018 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:09.021+0000 7f84f57f2540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:09.025 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:09.025 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.50 -s 1 2023-12-24T17:59:09.052 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:09.055+0000 7f19617f6540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:09.062 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:09.063 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.51 -s 1 2023-12-24T17:59:09.091 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:09.093+0000 7fa5565af540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:09.102 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:09.102 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.52 -s 1 2023-12-24T17:59:09.131 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:09.133+0000 7f73b5f49540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:09.136 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:09.137 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.53 -s 1 2023-12-24T17:59:09.165 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:09.168+0000 7f555aca8540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:09.171 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:09.171 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.54 -s 1 2023-12-24T17:59:09.199 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:09.201+0000 7fdfea312540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:09.204 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:09.204 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.55 -s 1 2023-12-24T17:59:09.233 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:09.235+0000 7fb179a2b540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:09.238 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:09.238 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.56 -s 1 2023-12-24T17:59:09.266 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:09.268+0000 7f3e5c75e540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:09.273 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:09.273 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.57 -s 1 2023-12-24T17:59:09.300 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:09.302+0000 7f5bcbb8b540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:09.311 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:09.311 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.58 -s 1 2023-12-24T17:59:09.340 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:09.342+0000 7fba3c658540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:09.347 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:09.347 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.59 -s 1 2023-12-24T17:59:09.375 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:09.377+0000 7f95d1dc0540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:09.381 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:09.381 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.60 -s 1 2023-12-24T17:59:09.409 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:09.411+0000 7f4b137b3540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:09.415 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:09.415 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.61 -s 1 2023-12-24T17:59:09.443 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:09.445+0000 7f0ad5fab540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:09.450 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:09.451 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.62 -s 1 2023-12-24T17:59:09.479 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:09.481+0000 7f9e3cd24540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:09.489 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:09.489 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.63 -s 1 2023-12-24T17:59:09.519 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:09.519+0000 7fbdad559540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:09.529 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:09.529 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.64 -s 1 2023-12-24T17:59:09.555 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:09.558+0000 7f342ba9f540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:09.566 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:09.566 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.65 -s 1 2023-12-24T17:59:09.596 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:09.599+0000 7f764242b540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:09.603 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:09.603 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.66 -s 1 2023-12-24T17:59:09.630 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:09.633+0000 7fd3f4e45540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:09.637 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:09.637 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.67 -s 1 2023-12-24T17:59:09.665 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:09.667+0000 7f64dc18e540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:09.671 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:09.671 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.68 -s 1 2023-12-24T17:59:09.699 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:09.701+0000 7f3e57702540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:09.704 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:09.705 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.69 -s 1 2023-12-24T17:59:09.732 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:09.735+0000 7f5d444af540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:09.739 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:09.739 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.70 -s 1 2023-12-24T17:59:09.767 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:09.769+0000 7f6abf783540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:09.775 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:09.775 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.71 -s 1 2023-12-24T17:59:09.803 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:09.806+0000 7fba6210e540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:09.810 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:09.810 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.72 -s 1 2023-12-24T17:59:09.839 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:09.841+0000 7f061290f540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:09.854 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:09.854 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.73 -s 1 2023-12-24T17:59:09.881 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:09.884+0000 7f0345f73540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:09.893 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:09.893 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.74 -s 1 2023-12-24T17:59:09.921 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:09.923+0000 7f492fcf7540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:09.931 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:09.931 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.75 -s 1 2023-12-24T17:59:09.960 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:09.962+0000 7f7aa1742540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:09.971 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:09.971 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.76 -s 1 2023-12-24T17:59:09.998 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:10.001+0000 7fe2da6d4540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:10.006 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:10.007 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.77 -s 1 2023-12-24T17:59:10.035 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:10.037+0000 7f97d1495540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:10.047 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:10.047 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.78 -s 1 2023-12-24T17:59:10.075 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:10.078+0000 7fd8a879b540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:10.085 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:10.085 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.79 -s 1 2023-12-24T17:59:10.112 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:10.115+0000 7f8882191540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:10.118 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:10.118 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.80 -s 1 2023-12-24T17:59:10.146 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:10.149+0000 7fbef0fa7540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:10.161 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:10.161 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.81 -s 1 2023-12-24T17:59:10.188 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:10.191+0000 7f572d074540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:10.198 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:10.198 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.82 -s 1 2023-12-24T17:59:10.225 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:10.228+0000 7f03a133e540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:10.232 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:10.233 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.83 -s 1 2023-12-24T17:59:10.261 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:10.263+0000 7ff3e5422540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:10.268 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:10.269 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.84 -s 1 2023-12-24T17:59:10.297 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:10.299+0000 7fe060e42540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:10.305 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:10.305 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.85 -s 1 2023-12-24T17:59:10.332 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:10.335+0000 7fc24e972540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:10.346 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:10.346 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.86 -s 1 2023-12-24T17:59:10.374 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:10.377+0000 7f39456fe540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:10.381 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:10.382 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.87 -s 1 2023-12-24T17:59:10.409 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:10.412+0000 7f906f32e540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:10.416 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:10.416 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.88 -s 1 2023-12-24T17:59:10.444 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:10.447+0000 7f1e332a7540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:10.450 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:10.450 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.89 -s 1 2023-12-24T17:59:10.478 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:10.480+0000 7fa0e0412540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:10.485 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:10.486 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.90 -s 1 2023-12-24T17:59:10.513 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:10.516+0000 7f36d9d4c540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:10.520 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:10.521 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.91 -s 1 2023-12-24T17:59:10.548 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:10.551+0000 7f0f62643540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:10.554 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:10.554 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.92 -s 1 2023-12-24T17:59:10.582 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:10.584+0000 7f13044be540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:10.588 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:10.588 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.93 -s 1 2023-12-24T17:59:10.615 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:10.618+0000 7f0f46298540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:10.626 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:10.627 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.94 -s 1 2023-12-24T17:59:10.654 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:10.657+0000 7ffbeed03540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:10.662 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:10.663 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.95 -s 1 2023-12-24T17:59:10.690 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:10.693+0000 7f68488bf540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:10.697 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:10.697 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.96 -s 1 2023-12-24T17:59:10.725 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:10.727+0000 7fb0351ed540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:10.733 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:10.733 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.97 -s 1 2023-12-24T17:59:10.760 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:10.763+0000 7f3d10d20540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:10.768 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:10.768 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.98 -s 1 2023-12-24T17:59:10.796 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:10.799+0000 7fe2eb036540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:10.809 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:10.809 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.99 -s 1 2023-12-24T17:59:10.837 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:10.839+0000 7f0abab6c540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:10.848 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T17:59:10.848 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 100 2023-12-24T17:59:10.848 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T17:59:10.875 INFO:tasks.workunit.client.0.smithi087.stdout:100 2023-12-24T17:59:10.897 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls -l 2023-12-24T17:59:10.898 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep image 2023-12-24T17:59:10.898 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T17:59:10.898 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 100 2023-12-24T17:59:10.946 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:10.948+0000 7fc81b238700 0 -- 172.21.15.87:0/1373706844 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x55cf5a7a4f10 msgr2=0x55cf5a7ab080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:10.954 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:10.956+0000 7fc81b238700 0 -- 172.21.15.87:0/1373706844 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fc80003a190 msgr2=0x7fc80003c5b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:11.127 INFO:tasks.workunit.client.0.smithi087.stdout:100 2023-12-24T17:59:11.127 INFO:tasks.workunit.client.0.smithi087.stderr:++ seq -w 00 99 2023-12-24T17:59:11.128 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:11.128 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.00 2023-12-24T17:59:11.167 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:11.169 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:11.170 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.01 2023-12-24T17:59:11.209 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:11.211 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:11.212 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.02 2023-12-24T17:59:11.252 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:11.255 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:11.255 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.03 2023-12-24T17:59:11.496 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:11.498 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:11.498 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.04 2023-12-24T17:59:11.534 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:11.537 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:11.537 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.05 2023-12-24T17:59:11.584 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:11.587 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:11.587 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.06 2023-12-24T17:59:11.624 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:11.627 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:11.627 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.07 2023-12-24T17:59:11.662 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:11.664 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:11.664 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.08 2023-12-24T17:59:11.707 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:11.709 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:11.709 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.09 2023-12-24T17:59:11.747 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:11.749 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:11.750 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.10 2023-12-24T17:59:11.784 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:11.787 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:11.787 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.11 2023-12-24T17:59:11.822 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:11.824 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:11.824 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.12 2023-12-24T17:59:11.861 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:11.863 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:11.863 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.13 2023-12-24T17:59:11.901 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:11.903 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:11.904 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.14 2023-12-24T17:59:11.939 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:11.941 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:11.941 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.15 2023-12-24T17:59:11.976 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:11.979 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:11.979 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.16 2023-12-24T17:59:12.014 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:12.016 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:12.016 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.17 2023-12-24T17:59:12.052 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:12.054 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:12.054 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.18 2023-12-24T17:59:12.090 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:12.092 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:12.092 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.19 2023-12-24T17:59:12.127 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:12.129 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:12.129 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.20 2023-12-24T17:59:12.166 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:12.168 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:12.168 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.21 2023-12-24T17:59:12.205 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:12.208 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:12.208 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.22 2023-12-24T17:59:12.255 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:12.257 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:12.257 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.23 2023-12-24T17:59:12.298 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:12.301 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:12.301 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.24 2023-12-24T17:59:12.341 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:12.345 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:12.346 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.25 2023-12-24T17:59:12.384 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:12.386 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:12.386 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.26 2023-12-24T17:59:12.424 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:12.426 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:12.427 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.27 2023-12-24T17:59:12.468 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:12.470 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:12.471 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.28 2023-12-24T17:59:12.509 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:12.511 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:12.511 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.29 2023-12-24T17:59:12.553 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:12.556 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:12.556 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.30 2023-12-24T17:59:12.598 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:12.600 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:12.600 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.31 2023-12-24T17:59:12.638 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:12.641 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:12.641 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.32 2023-12-24T17:59:12.676 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:12.679 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:12.679 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.33 2023-12-24T17:59:12.714 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:12.716 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:12.716 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.34 2023-12-24T17:59:12.752 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:12.754 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:12.754 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.35 2023-12-24T17:59:12.790 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:12.793 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:12.793 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.36 2023-12-24T17:59:12.829 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:12.831 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:12.831 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.37 2023-12-24T17:59:12.867 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:12.870 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:12.870 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.38 2023-12-24T17:59:12.907 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:12.909 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:12.909 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.39 2023-12-24T17:59:12.951 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:12.953 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:12.953 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.40 2023-12-24T17:59:12.993 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:12.995 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:12.995 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.41 2023-12-24T17:59:13.033 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:13.035 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:13.035 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.42 2023-12-24T17:59:13.071 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:13.073 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:13.074 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.43 2023-12-24T17:59:13.115 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:13.117 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:13.117 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.44 2023-12-24T17:59:13.159 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:13.161 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:13.161 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.45 2023-12-24T17:59:13.205 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:13.208 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:13.208 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.46 2023-12-24T17:59:13.246 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:13.248 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:13.248 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.47 2023-12-24T17:59:13.289 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:13.292 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:13.292 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.48 2023-12-24T17:59:13.328 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:13.330 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:13.330 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.49 2023-12-24T17:59:13.370 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:13.370 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:13.370 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.50 2023-12-24T17:59:13.404 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:13.406 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:13.406 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.51 2023-12-24T17:59:13.441 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:13.444 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:13.444 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.52 2023-12-24T17:59:13.481 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:13.483 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:13.483 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.53 2023-12-24T17:59:13.533 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:13.535 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:13.536 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.54 2023-12-24T17:59:13.578 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:13.581 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:13.581 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.55 2023-12-24T17:59:13.619 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:13.622 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:13.622 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.56 2023-12-24T17:59:13.657 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:13.660 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:13.660 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.57 2023-12-24T17:59:13.696 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:13.698 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:13.698 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.58 2023-12-24T17:59:13.737 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:13.739 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:13.739 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.59 2023-12-24T17:59:13.782 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:13.784 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:13.784 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.60 2023-12-24T17:59:13.824 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:13.826 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:13.826 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.61 2023-12-24T17:59:13.865 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:13.867 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:13.867 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.62 2023-12-24T17:59:13.915 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:13.917 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:13.917 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.63 2023-12-24T17:59:13.967 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:13.969 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:13.969 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.64 2023-12-24T17:59:14.006 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:14.008 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:14.008 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.65 2023-12-24T17:59:14.060 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:14.062 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:14.062 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.66 2023-12-24T17:59:14.099 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:14.101 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:14.101 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.67 2023-12-24T17:59:14.139 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:14.141 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:14.141 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.68 2023-12-24T17:59:14.176 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:14.179 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:14.179 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.69 2023-12-24T17:59:14.214 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:14.216 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:14.217 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.70 2023-12-24T17:59:14.252 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:14.254 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:14.254 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.71 2023-12-24T17:59:14.294 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:14.297 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:14.297 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.72 2023-12-24T17:59:14.336 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:14.339 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:14.339 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.73 2023-12-24T17:59:14.382 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:14.384 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:14.384 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.74 2023-12-24T17:59:14.422 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:14.425 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:14.425 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.75 2023-12-24T17:59:14.466 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:14.468 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:14.468 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.76 2023-12-24T17:59:14.506 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:14.509 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:14.509 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.77 2023-12-24T17:59:14.555 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:14.558 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:14.558 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.78 2023-12-24T17:59:14.597 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:14.599 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:14.599 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.79 2023-12-24T17:59:14.642 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:14.645 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:14.645 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.80 2023-12-24T17:59:14.685 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:14.687 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:14.687 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.81 2023-12-24T17:59:14.727 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:14.729 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:14.730 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.82 2023-12-24T17:59:14.784 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:14.787 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:14.787 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.83 2023-12-24T17:59:14.823 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:14.826 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:14.826 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.84 2023-12-24T17:59:14.861 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:14.864 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:14.864 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.85 2023-12-24T17:59:14.900 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:14.902 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:14.903 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.86 2023-12-24T17:59:14.938 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:14.941 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:14.941 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.87 2023-12-24T17:59:14.977 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:14.979 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:14.979 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.88 2023-12-24T17:59:15.015 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:15.017 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:15.017 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.89 2023-12-24T17:59:15.055 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:15.056 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:15.056 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.90 2023-12-24T17:59:15.098 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:15.100 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:15.100 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.91 2023-12-24T17:59:15.137 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:15.140 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:15.140 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.92 2023-12-24T17:59:15.180 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:15.183 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:15.183 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.93 2023-12-24T17:59:15.228 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:15.230 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:15.230 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.94 2023-12-24T17:59:15.268 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:15.270 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:15.270 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.95 2023-12-24T17:59:15.305 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:15.307 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:15.308 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.96 2023-12-24T17:59:15.349 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:15.351 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:15.351 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.97 2023-12-24T17:59:15.393 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:15.396 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:15.396 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.98 2023-12-24T17:59:15.442 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:15.445 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:15.445 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.99 2023-12-24T17:59:15.487 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:15.489 INFO:tasks.workunit.client.0.smithi087.stderr:++ seq -w 00 99 2023-12-24T17:59:15.491 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:15.491 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.00 --image-format 2 -s 1 2023-12-24T17:59:15.532 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:15.533 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.01 --image-format 2 -s 1 2023-12-24T17:59:15.574 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:15.574 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.02 --image-format 2 -s 1 2023-12-24T17:59:15.616 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:15.616 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.03 --image-format 2 -s 1 2023-12-24T17:59:15.657 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:15.657 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.04 --image-format 2 -s 1 2023-12-24T17:59:15.704 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:15.704 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.05 --image-format 2 -s 1 2023-12-24T17:59:15.756 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:15.756 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.06 --image-format 2 -s 1 2023-12-24T17:59:15.802 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:15.802 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.07 --image-format 2 -s 1 2023-12-24T17:59:15.848 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:15.848 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.08 --image-format 2 -s 1 2023-12-24T17:59:15.890 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:15.890 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.09 --image-format 2 -s 1 2023-12-24T17:59:15.935 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:15.935 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.10 --image-format 2 -s 1 2023-12-24T17:59:15.976 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:15.976 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.11 --image-format 2 -s 1 2023-12-24T17:59:16.017 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:16.017 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.12 --image-format 2 -s 1 2023-12-24T17:59:16.064 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:16.065 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.13 --image-format 2 -s 1 2023-12-24T17:59:16.105 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:16.105 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.14 --image-format 2 -s 1 2023-12-24T17:59:16.147 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:16.147 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.15 --image-format 2 -s 1 2023-12-24T17:59:16.196 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:16.196 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.16 --image-format 2 -s 1 2023-12-24T17:59:16.245 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:16.245 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.17 --image-format 2 -s 1 2023-12-24T17:59:16.287 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:16.287 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.18 --image-format 2 -s 1 2023-12-24T17:59:16.341 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:16.341 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.19 --image-format 2 -s 1 2023-12-24T17:59:16.387 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:16.387 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.20 --image-format 2 -s 1 2023-12-24T17:59:16.432 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:16.432 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.21 --image-format 2 -s 1 2023-12-24T17:59:16.477 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:16.478 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.22 --image-format 2 -s 1 2023-12-24T17:59:16.519 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:16.519 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.23 --image-format 2 -s 1 2023-12-24T17:59:16.562 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:16.562 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.24 --image-format 2 -s 1 2023-12-24T17:59:16.603 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:16.604 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.25 --image-format 2 -s 1 2023-12-24T17:59:16.652 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:16.652 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.26 --image-format 2 -s 1 2023-12-24T17:59:16.700 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:16.700 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.27 --image-format 2 -s 1 2023-12-24T17:59:16.742 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:16.743 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.28 --image-format 2 -s 1 2023-12-24T17:59:16.796 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:16.796 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.29 --image-format 2 -s 1 2023-12-24T17:59:16.842 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:16.842 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.30 --image-format 2 -s 1 2023-12-24T17:59:16.895 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:16.896 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.31 --image-format 2 -s 1 2023-12-24T17:59:16.938 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:16.938 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.32 --image-format 2 -s 1 2023-12-24T17:59:16.979 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:16.980 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.33 --image-format 2 -s 1 2023-12-24T17:59:17.019 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:17.019 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.34 --image-format 2 -s 1 2023-12-24T17:59:17.061 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:17.061 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.35 --image-format 2 -s 1 2023-12-24T17:59:17.104 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:17.105 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.36 --image-format 2 -s 1 2023-12-24T17:59:17.148 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:17.148 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.37 --image-format 2 -s 1 2023-12-24T17:59:17.192 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:17.192 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.38 --image-format 2 -s 1 2023-12-24T17:59:17.240 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:17.240 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.39 --image-format 2 -s 1 2023-12-24T17:59:17.294 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:17.294 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.40 --image-format 2 -s 1 2023-12-24T17:59:17.339 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:17.339 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.41 --image-format 2 -s 1 2023-12-24T17:59:17.388 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:17.388 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.42 --image-format 2 -s 1 2023-12-24T17:59:17.447 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:17.447 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.43 --image-format 2 -s 1 2023-12-24T17:59:17.488 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:17.489 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.44 --image-format 2 -s 1 2023-12-24T17:59:17.529 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:17.529 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.45 --image-format 2 -s 1 2023-12-24T17:59:17.571 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:17.571 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.46 --image-format 2 -s 1 2023-12-24T17:59:17.613 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:17.613 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.47 --image-format 2 -s 1 2023-12-24T17:59:17.655 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:17.656 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.48 --image-format 2 -s 1 2023-12-24T17:59:17.700 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:17.700 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.49 --image-format 2 -s 1 2023-12-24T17:59:17.750 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:17.750 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.50 --image-format 2 -s 1 2023-12-24T17:59:17.796 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:17.796 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.51 --image-format 2 -s 1 2023-12-24T17:59:17.844 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:17.844 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.52 --image-format 2 -s 1 2023-12-24T17:59:17.894 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:17.894 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.53 --image-format 2 -s 1 2023-12-24T17:59:17.935 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:17.935 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.54 --image-format 2 -s 1 2023-12-24T17:59:17.976 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:17.977 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.55 --image-format 2 -s 1 2023-12-24T17:59:18.018 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:18.018 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.56 --image-format 2 -s 1 2023-12-24T17:59:18.062 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:18.062 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.57 --image-format 2 -s 1 2023-12-24T17:59:18.106 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:18.106 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.58 --image-format 2 -s 1 2023-12-24T17:59:18.146 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:18.146 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.59 --image-format 2 -s 1 2023-12-24T17:59:18.193 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:18.193 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.60 --image-format 2 -s 1 2023-12-24T17:59:18.238 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:18.238 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.61 --image-format 2 -s 1 2023-12-24T17:59:18.284 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:18.284 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.62 --image-format 2 -s 1 2023-12-24T17:59:18.336 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:18.336 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.63 --image-format 2 -s 1 2023-12-24T17:59:18.381 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:18.381 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.64 --image-format 2 -s 1 2023-12-24T17:59:18.423 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:18.423 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.65 --image-format 2 -s 1 2023-12-24T17:59:18.463 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:18.464 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.66 --image-format 2 -s 1 2023-12-24T17:59:18.505 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:18.506 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.67 --image-format 2 -s 1 2023-12-24T17:59:18.548 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:18.548 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.68 --image-format 2 -s 1 2023-12-24T17:59:18.589 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:18.589 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.69 --image-format 2 -s 1 2023-12-24T17:59:18.630 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:18.630 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.70 --image-format 2 -s 1 2023-12-24T17:59:18.678 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:18.679 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.71 --image-format 2 -s 1 2023-12-24T17:59:18.722 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:18.722 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.72 --image-format 2 -s 1 2023-12-24T17:59:18.770 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:18.770 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.73 --image-format 2 -s 1 2023-12-24T17:59:18.820 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:18.820 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.74 --image-format 2 -s 1 2023-12-24T17:59:19.048 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:19.052+0000 7f51e93d1700 0 --2- 172.21.15.87:0/3948165775 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x562de4f60f80 0x562de4ff2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-12-24T17:59:19.069 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:19.069 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.75 --image-format 2 -s 1 2023-12-24T17:59:19.116 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:19.117 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.76 --image-format 2 -s 1 2023-12-24T17:59:19.161 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:19.161 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.77 --image-format 2 -s 1 2023-12-24T17:59:19.208 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:19.208 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.78 --image-format 2 -s 1 2023-12-24T17:59:19.250 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:19.250 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.79 --image-format 2 -s 1 2023-12-24T17:59:19.292 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:19.293 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.80 --image-format 2 -s 1 2023-12-24T17:59:19.335 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:19.335 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.81 --image-format 2 -s 1 2023-12-24T17:59:19.375 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:19.375 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.82 --image-format 2 -s 1 2023-12-24T17:59:19.418 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:19.418 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.83 --image-format 2 -s 1 2023-12-24T17:59:19.464 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:19.465 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.84 --image-format 2 -s 1 2023-12-24T17:59:19.505 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:19.505 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.85 --image-format 2 -s 1 2023-12-24T17:59:19.549 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:19.549 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.86 --image-format 2 -s 1 2023-12-24T17:59:19.595 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:19.595 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.87 --image-format 2 -s 1 2023-12-24T17:59:19.642 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:19.642 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.88 --image-format 2 -s 1 2023-12-24T17:59:19.688 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:19.688 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.89 --image-format 2 -s 1 2023-12-24T17:59:19.738 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:19.738 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.90 --image-format 2 -s 1 2023-12-24T17:59:19.783 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:19.783 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.91 --image-format 2 -s 1 2023-12-24T17:59:19.824 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:19.824 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.92 --image-format 2 -s 1 2023-12-24T17:59:19.865 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:19.866 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.93 --image-format 2 -s 1 2023-12-24T17:59:19.907 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:19.907 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.94 --image-format 2 -s 1 2023-12-24T17:59:19.948 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:19.948 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.95 --image-format 2 -s 1 2023-12-24T17:59:19.989 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:19.989 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.96 --image-format 2 -s 1 2023-12-24T17:59:20.032 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:20.033 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.97 --image-format 2 -s 1 2023-12-24T17:59:20.082 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:20.082 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.98 --image-format 2 -s 1 2023-12-24T17:59:20.124 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:20.124 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create image.99 --image-format 2 -s 1 2023-12-24T17:59:20.174 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T17:59:20.174 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T17:59:20.174 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 100 2023-12-24T17:59:20.201 INFO:tasks.workunit.client.0.smithi087.stdout:100 2023-12-24T17:59:20.202 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls -l 2023-12-24T17:59:20.202 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep image 2023-12-24T17:59:20.202 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T17:59:20.202 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 100 2023-12-24T17:59:20.255 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:20.258+0000 7fde037a4700 0 -- 172.21.15.87:0/3711844906 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x561ef5cb4d40 msgr2=0x561ef5cb7120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:20.264 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:20.267+0000 7fde037a4700 0 -- 172.21.15.87:0/3711844906 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fdde803a190 msgr2=0x7fdde803c5b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:20.540 INFO:tasks.workunit.client.0.smithi087.stdout:100 2023-12-24T17:59:20.540 INFO:tasks.workunit.client.0.smithi087.stderr:++ seq -w 00 99 2023-12-24T17:59:20.541 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:20.541 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.00 2023-12-24T17:59:20.651 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:20.655+0000 7fab9439f700 0 -- 172.21.15.87:0/327206459 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fab78008740 msgr2=0x7fab7800abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:20.652 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:20.654 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:20.655 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.01 2023-12-24T17:59:20.767 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:20.770+0000 7febca49f700 0 -- 172.21.15.87:0/1605667635 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7febb0008740 msgr2=0x7febb000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:20.767 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:20.770 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:20.770 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.02 2023-12-24T17:59:20.853 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:20.857+0000 7f752cc8c700 0 -- 172.21.15.87:0/319818300 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f7508006500 msgr2=0x7f7508011eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:20.854 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:20.856 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:20.856 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.03 2023-12-24T17:59:20.972 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:20.976+0000 7fb81b3c5700 0 -- 172.21.15.87:0/1245939343 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fb7f8008740 msgr2=0x7fb7f800abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:20.972 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:20.975 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:20.975 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.04 2023-12-24T17:59:21.076 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:21.080+0000 7fb886fb3700 0 -- 172.21.15.87:0/3917086498 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fb86c03a140 msgr2=0x7fb86c03c560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:21.086 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:21.089 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:21.089 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.05 2023-12-24T17:59:21.191 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:21.194+0000 7f50b72c2700 0 -- 172.21.15.87:0/3154376576 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f5098006760 msgr2=0x7f5098008bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:21.191 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:21.193 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:21.193 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.06 2023-12-24T17:59:21.282 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:21.286+0000 7f6141b2f700 0 -- 172.21.15.87:0/440449395 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7f612403a0c0 msgr2=0x7f612403c4e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:21.296 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:21.299 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:21.299 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.07 2023-12-24T17:59:21.412 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:21.415+0000 7f78ebeaa700 0 -- 172.21.15.87:0/2977029718 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f78c4005f80 msgr2=0x7f78c4008400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:21.412 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:21.415 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:21.415 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.08 2023-12-24T17:59:21.505 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:21.508+0000 7f655d16a700 0 -- 172.21.15.87:0/4219152685 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f6538008740 msgr2=0x7f653800abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:21.505 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:21.508 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:21.508 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.09 2023-12-24T17:59:21.609 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:21.613+0000 7f76e4293700 0 -- 172.21.15.87:0/1811383942 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f76c8008740 msgr2=0x7f76c800abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-24T17:59:21.610 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:21.613 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:21.613 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.10 2023-12-24T17:59:21.705 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:21.708+0000 7fc80807e700 0 -- 172.21.15.87:0/4198105112 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fc7e4008740 msgr2=0x7fc7e400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:21.705 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:21.708 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:21.708 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.11 2023-12-24T17:59:21.808 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:21.812+0000 7f8ab083b700 0 -- 172.21.15.87:0/430243536 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f8a94008740 msgr2=0x7f8a9400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:21.809 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:21.811 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:21.812 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.12 2023-12-24T17:59:21.906 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:21.909+0000 7f8fdb3ee700 0 -- 172.21.15.87:0/2941616758 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x55e1c373f670 msgr2=0x55e1c373c7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:21.906 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:21.908 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:21.908 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.13 2023-12-24T17:59:22.000 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:22.003+0000 7fa1ca610700 0 -- 172.21.15.87:0/1864219793 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fa1b0006500 msgr2=0x7fa1b0011eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-24T17:59:22.000 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:22.003 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:22.003 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.14 2023-12-24T17:59:22.089 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:22.093+0000 7f52f0f73700 0 -- 172.21.15.87:0/2188303485 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f52cc008740 msgr2=0x7f52cc00abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-24T17:59:22.090 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:22.092 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:22.093 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.15 2023-12-24T17:59:22.191 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:22.195+0000 7f1b01486700 0 -- 172.21.15.87:0/1374183593 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f1ae0005fd0 msgr2=0x7f1ae0008450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:22.192 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:22.194 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:22.194 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.16 2023-12-24T17:59:22.298 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:22.301+0000 7fd67b90d700 0 -- 172.21.15.87:0/3974326987 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fd65c0067a0 msgr2=0x7fd65c008be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:22.298 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:22.300 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:22.301 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.17 2023-12-24T17:59:22.602 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:22.605+0000 7f1177610700 0 -- 172.21.15.87:0/812358078 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f1154008740 msgr2=0x7f115400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:22.607 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:22.609 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:22.609 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.18 2023-12-24T17:59:22.709 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:22.711+0000 7f11b6e97700 0 -- 172.21.15.87:0/3267428222 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f1194008740 msgr2=0x7f119400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:22.709 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:22.710 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:22.710 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.19 2023-12-24T17:59:22.822 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:22.826+0000 7f148e7cc700 0 -- 172.21.15.87:0/72415504 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f14740065b0 msgr2=0x7f14740128d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:22.824 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:22.826 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:22.826 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.20 2023-12-24T17:59:22.912 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:22.915+0000 7fdefc7e2700 0 -- 172.21.15.87:0/385001480 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fdee0006500 msgr2=0x7fdee0011eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:22.912 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:22.915 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:22.915 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.21 2023-12-24T17:59:23.013 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:23.016+0000 7f66f1a7c700 0 -- 172.21.15.87:0/1027514232 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f66d8008740 msgr2=0x7f66d800abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:23.013 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:23.016 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:23.016 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.22 2023-12-24T17:59:23.104 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:23.108+0000 7f7230f83700 0 -- 172.21.15.87:0/4082652195 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f720c008740 msgr2=0x7f720c00abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-24T17:59:23.105 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:23.107 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:23.108 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.23 2023-12-24T17:59:23.210 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:23.213 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:23.213 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.24 2023-12-24T17:59:23.314 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:23.318+0000 7f188e08d700 0 -- 172.21.15.87:0/4177627787 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f1874008740 msgr2=0x7f187400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:23.314 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:23.317 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:23.317 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.25 2023-12-24T17:59:23.403 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:23.406+0000 7f8e1067e700 0 -- 172.21.15.87:0/3902981030 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f8dec008740 msgr2=0x7f8dec00abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:23.403 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:23.406 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:23.406 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.26 2023-12-24T17:59:23.499 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:23.503+0000 7fc45e9d5700 0 -- 172.21.15.87:0/6376277 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fc43c008740 msgr2=0x7fc43c00abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:23.500 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:23.502 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:23.502 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.27 2023-12-24T17:59:23.587 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:23.590+0000 7f428f32a700 0 -- 172.21.15.87:0/1829814968 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f4268005fd0 msgr2=0x7f4268008460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:23.587 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:23.589 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:23.589 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.28 2023-12-24T17:59:23.704 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:23.707+0000 7f3732794700 0 -- 172.21.15.87:0/1213114438 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f370c005ff0 msgr2=0x7f370c008470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:23.704 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:23.707 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:23.707 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.29 2023-12-24T17:59:23.791 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:23.795+0000 7f969a1ee700 0 -- 172.21.15.87:0/2310563174 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x5600c7a1db50 msgr2=0x5600c7ac3b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:23.792 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:23.795 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:23.795 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.30 2023-12-24T17:59:23.891 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:23.894+0000 7f7253b6b700 0 -- 172.21.15.87:0/252268841 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f7230008740 msgr2=0x7f723000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:23.891 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:23.894 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:23.894 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.31 2023-12-24T17:59:24.008 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:24.012+0000 7f36e7b4e700 0 -- 172.21.15.87:0/1716008884 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x55d5a6c27980 msgr2=0x55d5a6cce720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:24.009 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:24.012 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:24.012 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.32 2023-12-24T17:59:24.101 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:24.104+0000 7f445bb53700 0 -- 172.21.15.87:0/1017070859 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f4438008740 msgr2=0x7f443800abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:24.101 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:24.104 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:24.104 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.33 2023-12-24T17:59:24.192 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:24.195+0000 7f0b829bf700 0 -- 172.21.15.87:0/1313284124 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f0b64006720 msgr2=0x7f0b64008ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:24.192 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:24.195 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:24.195 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.34 2023-12-24T17:59:24.301 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:24.304+0000 7fae9c427700 0 -- 172.21.15.87:0/805684371 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fae80008740 msgr2=0x7fae8000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:24.301 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:24.304 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:24.304 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.35 2023-12-24T17:59:24.461 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:24.464+0000 7f2d1c690700 0 -- 172.21.15.87:0/2923726837 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x55b254228ca0 msgr2=0x55b25422caa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:24.461 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:24.463 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:24.464 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.36 2023-12-24T17:59:24.555 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:24.559+0000 7f9ae29b5700 0 -- 172.21.15.87:0/974367013 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x55f7eab2ab50 msgr2=0x55f7eabd0660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:24.557 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:24.559 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:24.560 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.37 2023-12-24T17:59:24.664 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:24.668+0000 7f10f87ba700 0 -- 172.21.15.87:0/3897892385 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f10d4008740 msgr2=0x7f10d400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:24.664 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:24.667 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:24.667 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.38 2023-12-24T17:59:24.791 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:24.794+0000 7fddb702f700 0 -- 172.21.15.87:0/1534142301 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fdd9c008740 msgr2=0x7fdd9c00abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:24.791 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:24.793 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:24.793 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.39 2023-12-24T17:59:24.885 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:24.888+0000 7fa9f621a700 0 -- 172.21.15.87:0/3984021064 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fa9dc008740 msgr2=0x7fa9dc00abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:24.885 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:24.888 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:24.888 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.40 2023-12-24T17:59:25.002 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:25.005+0000 7f0452486700 0 -- 172.21.15.87:0/2074772274 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f042c005fb0 msgr2=0x7f042c008460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:25.002 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:25.005 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:25.005 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.41 2023-12-24T17:59:25.116 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:25.120+0000 7f0cb70af700 0 -- 172.21.15.87:0/133833419 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f0c9c0087b0 msgr2=0x7f0c9c011ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:25.117 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:25.119 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:25.119 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.42 2023-12-24T17:59:25.238 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:25.241+0000 7f5ea8bdb700 0 -- 172.21.15.87:0/346843446 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f5e840064c0 msgr2=0x7f5e84011e80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:25.238 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:25.240 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:25.241 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.43 2023-12-24T17:59:25.327 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:25.331+0000 7f797164e700 0 -- 172.21.15.87:0/4266652848 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f7950005fb0 msgr2=0x7f7950008460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:25.328 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:25.330 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:25.331 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.44 2023-12-24T17:59:25.468 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:25.472+0000 7fe4fb560700 0 -- 172.21.15.87:0/3118521498 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fe4d8008740 msgr2=0x7fe4d800abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:25.469 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:25.472 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:25.472 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.45 2023-12-24T17:59:25.551 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:25.555+0000 7f9c8a981700 0 -- 172.21.15.87:0/1998927411 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7f9c6c03a260 msgr2=0x7f9c6c03c680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:25.560 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:25.563 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:25.563 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.46 2023-12-24T17:59:25.653 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:25.657+0000 7f704c22c700 0 -- 172.21.15.87:0/3959969131 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7f703003a190 msgr2=0x7f703003c5b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-24T17:59:25.662 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:25.665 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:25.665 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.47 2023-12-24T17:59:25.775 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:25.778+0000 7fbf18b16700 0 -- 172.21.15.87:0/3264900271 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fbefc0064c0 msgr2=0x7fbefc014d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:25.775 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:25.778 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:25.778 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.48 2023-12-24T17:59:25.869 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:25.870+0000 7f18a6e17700 0 -- 172.21.15.87:0/1838703621 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f1884008740 msgr2=0x7f188400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:25.869 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:25.869 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:25.870 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.49 2023-12-24T17:59:25.990 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:25.994+0000 7f6aa75d8700 0 -- 172.21.15.87:0/2268703764 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f6a84008740 msgr2=0x7f6a8400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-24T17:59:25.992 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:25.995 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:25.995 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.50 2023-12-24T17:59:26.119 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:26.122+0000 7f3226c32700 0 -- 172.21.15.87:0/33523471 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f3204008740 msgr2=0x7f320400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:26.119 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:26.122 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:26.122 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.51 2023-12-24T17:59:26.215 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:26.218+0000 7fc31f2ee700 0 -- 172.21.15.87:0/4208008629 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fc2fc008740 msgr2=0x7fc2fc00abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:26.215 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:26.218 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:26.218 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.52 2023-12-24T17:59:26.331 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:26.334+0000 7fa14c110700 0 -- 172.21.15.87:0/2112238069 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fa128008740 msgr2=0x7fa12800abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:26.331 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:26.333 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:26.334 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.53 2023-12-24T17:59:26.445 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:26.449+0000 7f40bc993700 0 -- 172.21.15.87:0/2594088093 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x55b7565b2f80 msgr2=0x55b756569370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:26.446 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:26.449 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:26.450 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.54 2023-12-24T17:59:26.580 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:26.583+0000 7f56b1700700 0 -- 172.21.15.87:0/3286399416 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x55f27475aa60 msgr2=0x55f27477cc00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:26.591 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:26.595 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:26.595 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.55 2023-12-24T17:59:26.708 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:26.711+0000 7f557dd26700 0 -- 172.21.15.87:0/1967439298 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f555c008740 msgr2=0x7f555c00abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:26.708 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:26.710 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:26.711 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.56 2023-12-24T17:59:26.826 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:26.829+0000 7f708e0d0700 0 -- 172.21.15.87:0/799975900 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f7068005fb0 msgr2=0x7f7068008460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:26.827 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:26.829 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:26.829 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.57 2023-12-24T17:59:26.920 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:26.924+0000 7f7e95694700 0 -- 172.21.15.87:0/364547451 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f7e70008740 msgr2=0x7f7e7000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:26.921 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:26.923 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:26.923 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.58 2023-12-24T17:59:27.018 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:27.022+0000 7f2643896700 0 -- 172.21.15.87:0/3843467627 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f26200065b0 msgr2=0x7f2620011ee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:27.019 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:27.021 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:27.021 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.59 2023-12-24T17:59:27.123 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:27.127+0000 7f0b00136700 0 -- 172.21.15.87:0/2102872606 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x5561172c5a50 msgr2=0x5561172c4c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:27.124 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:27.126 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:27.127 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.60 2023-12-24T17:59:27.226 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:27.229+0000 7f9b9a6e3700 0 -- 172.21.15.87:0/684500897 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x55f506537980 msgr2=0x7f9b7c03f540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:27.242 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:27.244 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:27.244 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.61 2023-12-24T17:59:27.339 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:27.342+0000 7f13570d9700 0 -- 172.21.15.87:0/2170132611 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f1334008740 msgr2=0x7f133400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-24T17:59:27.339 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:27.342 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:27.342 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.62 2023-12-24T17:59:27.437 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:27.440+0000 7fda8200c700 0 -- 172.21.15.87:0/1977924519 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fda68008740 msgr2=0x7fda6800abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:27.437 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:27.440 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:27.440 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.63 2023-12-24T17:59:27.550 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:27.554+0000 7fb461923700 0 -- 172.21.15.87:0/2516215298 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fb448008740 msgr2=0x7fb44800abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:27.550 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:27.553 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:27.553 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.64 2023-12-24T17:59:27.640 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:27.644+0000 7f6ead1b9700 0 -- 172.21.15.87:0/71590513 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f6e88006500 msgr2=0x7f6e88011eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:27.641 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:27.643 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:27.644 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.65 2023-12-24T17:59:27.727 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:27.731+0000 7fb6bbe28700 0 -- 172.21.15.87:0/2151697079 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fb6a0008740 msgr2=0x7fb6a000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:27.742 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:27.745 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:27.745 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.66 2023-12-24T17:59:27.844 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:27.848+0000 7f0b6ade8700 0 -- 172.21.15.87:0/3360101539 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x560c3f96e910 msgr2=0x560c3f9900a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:27.845 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:27.848 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:27.848 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.67 2023-12-24T17:59:27.940 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:27.942+0000 7f88b0462700 0 -- 172.21.15.87:0/928225645 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x560fd2bebf40 msgr2=0x560fd2bc9650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:27.940 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:27.943 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:27.943 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.68 2023-12-24T17:59:28.036 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:28.038+0000 7fca11e46700 0 -- 172.21.15.87:0/3409828711 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fc9f8008740 msgr2=0x7fc9f800abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:28.036 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:28.038 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:28.039 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.69 2023-12-24T17:59:28.146 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:28.148+0000 7f1265534700 0 -- 172.21.15.87:0/3412796828 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x5622f15c3ac0 msgr2=0x5622f1579430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:28.146 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:28.150 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:28.150 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.70 2023-12-24T17:59:28.252 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:28.255+0000 7f67cf03b700 0 -- 172.21.15.87:0/3838152822 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x559c0b5753d0 msgr2=0x559c0b586890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:28.257 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:28.260 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:28.260 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.71 2023-12-24T17:59:28.373 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:28.375+0000 7ff392560700 0 -- 172.21.15.87:0/1001565515 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7ff37800a950 msgr2=0x7ff37800cdd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:28.373 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:28.375 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:28.375 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.72 2023-12-24T17:59:28.658 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:28.660+0000 7fb440e13700 0 -- 172.21.15.87:0/1273211699 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7fb42403a140 msgr2=0x7fb42403c560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:28.668 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:28.670 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:28.670 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.73 2023-12-24T17:59:28.759 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:28.761+0000 7ff497ebd700 0 -- 172.21.15.87:0/2402373102 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7ff474008740 msgr2=0x7ff47400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:28.759 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:28.762 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:28.762 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.74 2023-12-24T17:59:28.860 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:28.863+0000 7fd368ba8700 0 -- 172.21.15.87:0/2295607541 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fd344008740 msgr2=0x7fd34400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:28.869 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:28.872 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:28.872 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.75 2023-12-24T17:59:28.972 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:28.975+0000 7fa18ef30700 0 -- 172.21.15.87:0/315052549 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fa174006500 msgr2=0x7fa174011eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:28.973 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:28.975 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:28.975 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.76 2023-12-24T17:59:29.062 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:29.064+0000 7fb65a509700 0 -- 172.21.15.87:0/1593771321 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7fb63c03a140 msgr2=0x7fb63c03c560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:29.068 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:29.070 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:29.070 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.77 2023-12-24T17:59:29.162 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:29.165+0000 7f9514de5700 0 -- 172.21.15.87:0/3491727519 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f94f0008740 msgr2=0x7f94f000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:29.163 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:29.165 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:29.166 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.78 2023-12-24T17:59:29.255 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:29.257+0000 7fd505524700 0 -- 172.21.15.87:0/1551639695 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fd4e4005fb0 msgr2=0x7fd4e4008460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:29.255 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:29.258 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:29.258 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.79 2023-12-24T17:59:29.362 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:29.364+0000 7f33ff718700 0 -- 172.21.15.87:0/4151498164 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7f33e403a140 msgr2=0x7f33e403c560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:29.372 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:29.375 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:29.375 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.80 2023-12-24T17:59:29.507 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:29.509+0000 7fec948b7700 0 -- 172.21.15.87:0/1808753767 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fec70008740 msgr2=0x7fec7000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:29.507 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:29.509 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:29.509 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.81 2023-12-24T17:59:29.597 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:29.600+0000 7fe2e9bef700 0 -- 172.21.15.87:0/1001134802 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x55f01de6d710 msgr2=0x55f01de463c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-24T17:59:29.600 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:29.603 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:29.603 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.82 2023-12-24T17:59:29.727 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:29.730+0000 7f81a445a700 0 -- 172.21.15.87:0/2241425885 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f8180008740 msgr2=0x7f818000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:29.727 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:29.730 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:29.730 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.83 2023-12-24T17:59:29.830 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:29.833+0000 7f879e468700 0 -- 172.21.15.87:0/1251259185 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x563dfb0d1cd0 msgr2=0x563dfb0aa670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:29.831 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:29.833 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:29.833 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.84 2023-12-24T17:59:29.931 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:29.933+0000 7f2cbc874700 0 -- 172.21.15.87:0/882373837 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f2ca003a190 msgr2=0x7f2ca003c5b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:29.941 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:29.944 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:29.944 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.85 2023-12-24T17:59:30.053 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:30.055+0000 7f97ced79700 0 -- 172.21.15.87:0/3880387433 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f97ac008740 msgr2=0x7f97ac00abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:30.053 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:30.056 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:30.056 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.86 2023-12-24T17:59:30.145 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:30.148+0000 7f84d5af6700 0 -- 172.21.15.87:0/3958689121 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f84b0005fb0 msgr2=0x7f84b0008460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-24T17:59:30.147 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:30.150 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:30.150 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.87 2023-12-24T17:59:30.247 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:30.250+0000 7f6371495700 0 -- 172.21.15.87:0/3666342000 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f634c008740 msgr2=0x7f634c00abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:30.248 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:30.251 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:30.251 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.88 2023-12-24T17:59:30.346 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:30.349+0000 7fbcdad3f700 0 -- 172.21.15.87:0/1671512103 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7fbcbc03a180 msgr2=0x7fbcbc03c5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:30.354 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:30.357 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:30.357 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.89 2023-12-24T17:59:30.441 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:30.444+0000 7f3de517b700 0 -- 172.21.15.87:0/1309527844 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f3dc803a140 msgr2=0x7f3dc803c560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:30.447 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:30.450 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:30.450 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.90 2023-12-24T17:59:30.559 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:30.561+0000 7f4a4e7cf700 0 -- 172.21.15.87:0/3676767093 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7f4a3003a140 msgr2=0x7f4a3003c560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:30.565 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:30.568 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:30.569 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.91 2023-12-24T17:59:30.651 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:30.653+0000 7fb600d74700 0 -- 172.21.15.87:0/500470074 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fb5dc0064c0 msgr2=0x7fb5dc011e80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:30.651 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:30.654 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:30.654 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.92 2023-12-24T17:59:30.774 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:30.773+0000 7f2dcecf9700 0 -- 172.21.15.87:0/3645079712 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f2dac008740 msgr2=0x7f2dac00abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:30.774 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:30.774 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:30.774 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.93 2023-12-24T17:59:30.887 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:30.890+0000 7f329b544700 0 -- 172.21.15.87:0/1335810126 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x56499746de00 msgr2=0x564997420750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:30.887 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:30.890 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:30.890 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.94 2023-12-24T17:59:30.974 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:30.977+0000 7f380285f700 0 -- 172.21.15.87:0/2774899043 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7f37e403a140 msgr2=0x7f37e403c560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:30.981 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:30.983 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:30.984 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.95 2023-12-24T17:59:31.087 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:31.090+0000 7f0b277a3700 0 -- 172.21.15.87:0/4216291256 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x55e75e544980 msgr2=0x55e75e5ea7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:31.097 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:31.100 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:31.100 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.96 2023-12-24T17:59:31.206 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:31.209+0000 7f3cc150e700 0 -- 172.21.15.87:0/3640346500 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x56035616af00 msgr2=0x560356120460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:31.207 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:31.209 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:31.209 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.97 2023-12-24T17:59:31.321 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:31.323+0000 7fcbd09fe700 0 -- 172.21.15.87:0/3551356244 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x561693b1c9c0 msgr2=0x561693b08440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:31.321 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:31.323 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:31.323 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.98 2023-12-24T17:59:31.414 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:31.417+0000 7f64af76b700 0 -- 172.21.15.87:0/1987688262 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x559544e1c590 msgr2=0x559544df35c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:31.415 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:31.417 INFO:tasks.workunit.client.0.smithi087.stderr:+ for i in $(seq -w 00 99) 2023-12-24T17:59:31.417 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm image.99 2023-12-24T17:59:31.517 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:31.520+0000 7ff99fa77700 0 -- 172.21.15.87:0/178602602 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7ff98403a0b0 msgr2=0x7ff98403c4d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:31.526 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:31.529 INFO:tasks.workunit.client.0.smithi087.stderr:+ test_remove 2023-12-24T17:59:31.529 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'testing remove...' 2023-12-24T17:59:31.529 INFO:tasks.workunit.client.0.smithi087.stdout:testing remove... 2023-12-24T17:59:31.530 INFO:tasks.workunit.client.0.smithi087.stderr:+ remove_images 2023-12-24T17:59:31.530 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:31.609 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:31.694 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:31.775 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:31.850 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:31.927 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:32.000 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:32.077 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:32.150 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:32.225 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:32.300 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:32.375 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:32.451 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:32.525 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:32.609 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:32.684 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:32.757 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:32.830 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:32.903 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd remove NOT_EXIST 2023-12-24T17:59:32.944 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 0% complete...failed. 2023-12-24T17:59:32.944 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: delete error: (2) No such file or directory 2023-12-24T17:59:32.946 INFO:tasks.workunit.client.0.smithi087.stderr:+ true 2023-12-24T17:59:32.946 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 1 -s 1 test1 2023-12-24T17:59:32.970 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: image format 1 is deprecated 2023-12-24T17:59:32.977 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:32.979+0000 7f77b2e6b540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:32.983 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test1 2023-12-24T17:59:33.023 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:33.026 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T17:59:33.026 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T17:59:33.026 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^0$' 2023-12-24T17:59:33.053 INFO:tasks.workunit.client.0.smithi087.stdout:0 2023-12-24T17:59:33.053 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 -s 1 test2 2023-12-24T17:59:33.097 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test2 2023-12-24T17:59:33.192 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:33.195+0000 7fe44c4fb700 0 -- 172.21.15.87:0/3097540398 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fe428006480 msgr2=0x7fe428014d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:33.192 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:33.195 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T17:59:33.196 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T17:59:33.196 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^0$' 2023-12-24T17:59:33.222 INFO:tasks.workunit.client.0.smithi087.stdout:0 2023-12-24T17:59:33.222 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 1 -s 1 test1 2023-12-24T17:59:33.245 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: image format 1 is deprecated 2023-12-24T17:59:33.251 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:33.253+0000 7f01f9d45540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:33.259 INFO:tasks.workunit.client.0.smithi087.stderr:+ rados rm -p rbd test1.rbd 2023-12-24T17:59:33.288 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test1 2023-12-24T17:59:33.320 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:33.322 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T17:59:33.322 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T17:59:33.322 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^0$' 2023-12-24T17:59:33.349 INFO:tasks.workunit.client.0.smithi087.stdout:0 2023-12-24T17:59:33.350 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 0 ']' 2023-12-24T17:59:33.350 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 -s 1 test2 2023-12-24T17:59:33.391 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create test2@snap 2023-12-24T17:59:34.120 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T17:59:34.132 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap protect test2@snap 2023-12-24T17:59:34.174 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd clone test2@snap clone --rbd-default-clone-format 1 2023-12-24T17:59:34.237 INFO:tasks.workunit.client.0.smithi087.stderr:+ rados -p rbd rm rbd_children 2023-12-24T17:59:34.268 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm clone 2023-12-24T17:59:34.351 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:34.353+0000 7f71d1fb8700 0 -- 172.21.15.87:0/3514073066 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7f71b403a130 msgr2=0x7f71b403c510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:34.363 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:34.366 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T17:59:34.366 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep clone 2023-12-24T17:59:34.366 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T17:59:34.366 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^0$' 2023-12-24T17:59:34.392 INFO:tasks.workunit.client.0.smithi087.stdout:0 2023-12-24T17:59:34.392 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap unprotect test2@snap 2023-12-24T17:59:34.432 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap rm test2@snap 2023-12-24T17:59:35.118 INFO:tasks.workunit.client.0.smithi087.stderr: Removing snap: 100% complete...done. 2023-12-24T17:59:35.129 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test2 2023-12-24T17:59:35.282 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:35.284+0000 7f722fe73700 0 -- 172.21.15.87:0/3936776781 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f7208005f80 msgr2=0x7f7208008400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:35.282 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:35.286 INFO:tasks.workunit.client.0.smithi087.stderr:+ test_migration 2023-12-24T17:59:35.286 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'testing migration...' 2023-12-24T17:59:35.286 INFO:tasks.workunit.client.0.smithi087.stdout:testing migration... 2023-12-24T17:59:35.286 INFO:tasks.workunit.client.0.smithi087.stderr:+ remove_images 2023-12-24T17:59:35.286 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:35.361 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:35.437 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:35.517 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:35.608 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:35.682 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:35.755 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:35.837 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:35.914 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:35.992 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:36.070 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:36.145 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:36.217 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:36.293 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:36.366 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:36.448 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:36.527 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:36.603 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T17:59:36.680 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph osd pool create rbd2 8 2023-12-24T17:59:37.240 INFO:tasks.workunit.client.0.smithi087.stderr:pool 'rbd2' already exists 2023-12-24T17:59:37.249 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd pool init rbd2 2023-12-24T17:59:40.137 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 1 -s 128M test1 2023-12-24T17:59:40.162 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: image format 1 is deprecated 2023-12-24T17:59:40.171 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:40.173+0000 7f203f2d1540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:40.183 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info test1 2023-12-24T17:59:40.183 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'format: 1' 2023-12-24T17:59:40.216 INFO:tasks.workunit.client.0.smithi087.stdout: format: 1 2023-12-24T17:59:40.217 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration prepare test1 --image-format 2 2023-12-24T17:59:40.293 INFO:tasks.workunit.client.0.smithi087.stderr:++ get_migration_state test1 2023-12-24T17:59:40.293 INFO:tasks.workunit.client.0.smithi087.stderr:++ local image=test1 2023-12-24T17:59:40.293 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd --format xml status test1 2023-12-24T17:59:40.293 INFO:tasks.workunit.client.0.smithi087.stderr:++ xmlstarlet sel -t -v //status/migration/state 2023-12-24T17:59:40.367 INFO:tasks.workunit.client.0.smithi087.stderr:+ test prepared = prepared 2023-12-24T17:59:40.367 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info test1 2023-12-24T17:59:40.367 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'format: 2' 2023-12-24T17:59:40.404 INFO:tasks.workunit.client.0.smithi087.stdout: format: 2 2023-12-24T17:59:40.404 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test1 2023-12-24T17:59:40.438 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:40.440+0000 7f3878325540 -1 librbd::image::PreRemoveRequest: 0x559f604fa890 validate_image_removal: image in migration state - not removing 2023-12-24T17:59:40.439 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 0% complete...failed. 2023-12-24T17:59:40.439 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: error: image still has watchers 2023-12-24T17:59:40.440 INFO:tasks.workunit.client.0.smithi087.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-24T17:59:40.442 INFO:tasks.workunit.client.0.smithi087.stderr:+ true 2023-12-24T17:59:40.442 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration execute test1 2023-12-24T17:59:40.550 INFO:tasks.workunit.client.0.smithi087.stderr: Image migration: 3% complete... Image migration: 6% complete... Image migration: 9% complete... Image migration: 12% complete... Image migration: 15% complete... Image migration: 18% complete... Image migration: 21% complete... Image migration: 25% complete... Image migration: 28% complete... Image migration: 31% complete... Image migration: 34% complete... Image migration: 37% complete... Image migration: 40% complete... Image migration: 43% complete... Image migration: 46% complete... Image migration: 50% complete... Image migration: 53% complete... Image migration: 56% complete... Image migration: 59% complete... Image migration: 62% complete... Image migration: 65% complete... Image migration: 68% complete... Image migration: 71% complete... Image migration: 75% complete... Image migration: 78% complete... Image migration: 81% complete... Image migration: 84% complete... Image migration: 87% complete... Image migration: 90% complete...2023-12-24T17:59:40.552+0000 7ff4cbec2700 0 -- 172.21.15.87:0/1835223088 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7ff4b003a930 msgr2=0x7ff4b003cd50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:40.562 INFO:tasks.workunit.client.0.smithi087.stderr: Image migration: 93% complete... Image migration: 96% complete...2023-12-24T17:59:40.565+0000 7ff4cbec2700 0 -- 172.21.15.87:0/1835223088 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x5616c7009980 msgr2=0x7ff4b003f690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-24T17:59:40.590 INFO:tasks.workunit.client.0.smithi087.stderr: Image migration: 100% complete...done. 2023-12-24T17:59:40.594 INFO:tasks.workunit.client.0.smithi087.stderr:++ get_migration_state test1 2023-12-24T17:59:40.594 INFO:tasks.workunit.client.0.smithi087.stderr:++ local image=test1 2023-12-24T17:59:40.594 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd --format xml status test1 2023-12-24T17:59:40.594 INFO:tasks.workunit.client.0.smithi087.stderr:++ xmlstarlet sel -t -v //status/migration/state 2023-12-24T17:59:40.656 INFO:tasks.workunit.client.0.smithi087.stderr:+ test executed = executed 2023-12-24T17:59:40.656 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration commit test1 2023-12-24T17:59:40.740 INFO:tasks.workunit.client.0.smithi087.stderr: Commit image migration: 3% complete... Commit image migration: 6% complete... Commit image migration: 9% complete... Commit image migration: 12% complete... Commit image migration: 15% complete... Commit image migration: 18% complete... Commit image migration: 21% complete... Commit image migration: 25% complete... Commit image migration: 28% complete... Commit image migration: 31% complete... Commit image migration: 34% complete... Commit image migration: 37% complete... Commit image migration: 40% complete... Commit image migration: 43% complete... Commit image migration: 46% complete...2023-12-24T17:59:40.742+0000 7faf059be700 0 -- 172.21.15.87:0/4074569195 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7faeec008740 msgr2=0x7faeec00abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:40.754 INFO:tasks.workunit.client.0.smithi087.stderr: Commit image migration: 50% complete... Commit image migration: 53% complete... Commit image migration: 56% complete... Commit image migration: 59% complete... Commit image migration: 62% complete... Commit image migration: 65% complete... Commit image migration: 68% complete... Commit image migration: 71% complete... Commit image migration: 75% complete... Commit image migration: 78% complete... Commit image migration: 81% complete... Commit image migration: 84% complete... Commit image migration: 87% complete... Commit image migration: 90% complete... Commit image migration: 93% complete... Commit image migration: 96% complete...2023-12-24T17:59:40.757+0000 7faf059be700 0 -- 172.21.15.87:0/4074569195 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7faee803a8e0 msgr2=0x7faee803cd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:40.769 INFO:tasks.workunit.client.0.smithi087.stderr: Commit image migration: 100% complete...done. 2023-12-24T17:59:40.772 INFO:tasks.workunit.client.0.smithi087.stderr:+ get_migration_state test1 2023-12-24T17:59:40.772 INFO:tasks.workunit.client.0.smithi087.stderr:+ local image=test1 2023-12-24T17:59:40.772 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd --format xml status test1 2023-12-24T17:59:40.772 INFO:tasks.workunit.client.0.smithi087.stderr:+ xmlstarlet sel -t -v //status/migration/state 2023-12-24T17:59:40.808 INFO:tasks.workunit.client.0.smithi087.stderr:+ true 2023-12-24T17:59:40.808 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info test1 2023-12-24T17:59:40.808 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'features: .*layering' 2023-12-24T17:59:40.841 INFO:tasks.workunit.client.0.smithi087.stderr:+ true 2023-12-24T17:59:40.841 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration prepare test1 --image-feature layering,exclusive-lock,object-map,fast-diff,deep-flatten 2023-12-24T17:59:40.950 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info test1 2023-12-24T17:59:40.950 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'features: .*layering' 2023-12-24T17:59:40.989 INFO:tasks.workunit.client.0.smithi087.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, migrating 2023-12-24T17:59:40.989 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration execute test1 2023-12-24T17:59:41.057 INFO:tasks.workunit.client.0.smithi087.stderr: Image migration: 3% complete... Image migration: 6% complete... Image migration: 9% complete... Image migration: 12% complete... Image migration: 15% complete... Image migration: 18% complete... Image migration: 21% complete... Image migration: 25% complete... Image migration: 28% complete... Image migration: 31% complete... Image migration: 34% complete... Image migration: 37% complete... Image migration: 40% complete... Image migration: 43% complete... Image migration: 46% complete... Image migration: 50% complete... Image migration: 53% complete... Image migration: 56% complete... Image migration: 59% complete... Image migration: 62% complete... Image migration: 65% complete... Image migration: 68% complete... Image migration: 71% complete... Image migration: 75% complete... Image migration: 78% complete... Image migration: 81% complete... Image migration: 84% complete... Image migration: 87% complete... Image migration: 90% complete... Image migration: 93% complete... Image migration: 96% complete...2023-12-24T17:59:41.060+0000 7f69d3690700 0 -- 172.21.15.87:0/3446497290 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x55ecab987980 msgr2=0x55ecaba7c8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:41.069 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:41.072+0000 7f69d2e8f700 0 -- 172.21.15.87:0/3446497290 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7f69b803a920 msgr2=0x7f69b803cd40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-24T17:59:41.081 INFO:tasks.workunit.client.0.smithi087.stderr: Image migration: 100% complete...done. 2023-12-24T17:59:41.084 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration commit test1 2023-12-24T17:59:41.166 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:41.169+0000 7f0c1268c700 0 -- 172.21.15.87:0/1403833604 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7f0bf8008740 msgr2=0x7f0bf800abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:41.177 INFO:tasks.workunit.client.0.smithi087.stderr: Commit image migration: 3% complete... Commit image migration: 6% complete... Commit image migration: 9% complete... Commit image migration: 12% complete... Commit image migration: 15% complete... Commit image migration: 18% complete... Commit image migration: 21% complete... Commit image migration: 25% complete... Commit image migration: 28% complete... Commit image migration: 31% complete... Commit image migration: 34% complete... Commit image migration: 37% complete... Commit image migration: 40% complete... Commit image migration: 43% complete... Commit image migration: 46% complete...2023-12-24T17:59:41.179+0000 7f0c1268c700 0 -- 172.21.15.87:0/1403833604 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7f0bf403a8e0 msgr2=0x7f0bf403cd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:41.207 INFO:tasks.workunit.client.0.smithi087.stderr: Commit image migration: 50% complete... Commit image migration: 53% complete... Commit image migration: 56% complete... Commit image migration: 59% complete... Commit image migration: 62% complete... Commit image migration: 65% complete... Commit image migration: 68% complete... Commit image migration: 71% complete... Commit image migration: 75% complete... Commit image migration: 78% complete... Commit image migration: 81% complete... Commit image migration: 84% complete... Commit image migration: 87% complete... Commit image migration: 90% complete... Commit image migration: 93% complete... Commit image migration: 96% complete... Commit image migration: 100% complete...done. 2023-12-24T17:59:41.210 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration prepare test1 rbd2/test1 2023-12-24T17:59:41.308 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:41.310+0000 7f058ba78700 0 -- 172.21.15.87:0/3466076650 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7f0568008740 msgr2=0x7f056800abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:41.314 INFO:tasks.workunit.client.0.smithi087.stderr:++ get_migration_state rbd2/test1 2023-12-24T17:59:41.314 INFO:tasks.workunit.client.0.smithi087.stderr:++ local image=rbd2/test1 2023-12-24T17:59:41.314 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd --format xml status rbd2/test1 2023-12-24T17:59:41.314 INFO:tasks.workunit.client.0.smithi087.stderr:++ xmlstarlet sel -t -v //status/migration/state 2023-12-24T17:59:41.384 INFO:tasks.workunit.client.0.smithi087.stderr:+ test prepared = prepared 2023-12-24T17:59:41.384 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T17:59:41.384 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T17:59:41.384 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^0$' 2023-12-24T17:59:41.411 INFO:tasks.workunit.client.0.smithi087.stdout:0 2023-12-24T17:59:41.411 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd -p rbd2 ls 2023-12-24T17:59:41.411 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep test1 2023-12-24T17:59:41.441 INFO:tasks.workunit.client.0.smithi087.stdout:test1 2023-12-24T17:59:41.441 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration execute test1 2023-12-24T17:59:41.527 INFO:tasks.workunit.client.0.smithi087.stderr: Image migration: 3% complete... Image migration: 6% complete... Image migration: 9% complete... Image migration: 12% complete... Image migration: 15% complete... Image migration: 18% complete... Image migration: 21% complete... Image migration: 25% complete... Image migration: 28% complete... Image migration: 31% complete... Image migration: 34% complete... Image migration: 37% complete... Image migration: 40% complete... Image migration: 43% complete... Image migration: 46% complete... Image migration: 50% complete... Image migration: 53% complete... Image migration: 56% complete... Image migration: 59% complete... Image migration: 62% complete... Image migration: 65% complete... Image migration: 68% complete... Image migration: 71% complete... Image migration: 75% complete... Image migration: 78% complete... Image migration: 81% complete... Image migration: 84% complete... Image migration: 87% complete... Image migration: 90% complete... Image migration: 93% complete... Image migration: 96% complete...2023-12-24T17:59:41.529+0000 7f3a74e4e700 0 -- 172.21.15.87:0/989948690 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x55aabdae28a0 msgr2=0x55aabdae4c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:41.541 INFO:tasks.workunit.client.0.smithi087.stderr: Image migration: 100% complete...done. 2023-12-24T17:59:41.544 INFO:tasks.workunit.client.0.smithi087.stderr:++ get_migration_state rbd2/test1 2023-12-24T17:59:41.544 INFO:tasks.workunit.client.0.smithi087.stderr:++ local image=rbd2/test1 2023-12-24T17:59:41.544 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd --format xml status rbd2/test1 2023-12-24T17:59:41.544 INFO:tasks.workunit.client.0.smithi087.stderr:++ xmlstarlet sel -t -v //status/migration/state 2023-12-24T17:59:41.607 INFO:tasks.workunit.client.0.smithi087.stderr:+ test executed = executed 2023-12-24T17:59:41.607 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm rbd2/test1 2023-12-24T17:59:41.646 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:41.648+0000 7f5327fff700 -1 librbd::image::PreRemoveRequest: 0x5597deb7aad0 validate_image_removal: image in migration state - not removing 2023-12-24T17:59:41.655 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 0% complete...failed. 2023-12-24T17:59:41.655 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: error: image still has watchers 2023-12-24T17:59:41.656 INFO:tasks.workunit.client.0.smithi087.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-24T17:59:41.658 INFO:tasks.workunit.client.0.smithi087.stderr:+ true 2023-12-24T17:59:41.658 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration commit test1 2023-12-24T17:59:41.949 INFO:tasks.workunit.client.0.smithi087.stderr: Commit image migration: 3% complete... Commit image migration: 6% complete... Commit image migration: 9% complete... Commit image migration: 12% complete... Commit image migration: 15% complete... Commit image migration: 18% complete... Commit image migration: 21% complete... Commit image migration: 25% complete... Commit image migration: 28% complete... Commit image migration: 31% complete... Commit image migration: 34% complete... Commit image migration: 37% complete... Commit image migration: 40% complete... Commit image migration: 43% complete... Commit image migration: 46% complete... Commit image migration: 50% complete... Commit image migration: 53% complete... Commit image migration: 56% complete... Commit image migration: 59% complete... Commit image migration: 62% complete... Commit image migration: 65% complete... Commit image migration: 68% complete... Commit image migration: 71% complete... Commit image migration: 75% complete... Commit image migration: 78% complete... Commit image migration: 81% complete... Commit image migration: 84% complete... Commit image migration: 87% complete... Commit image migration: 90% complete... Commit image migration: 93% complete... Commit image migration: 96% complete...2023-12-24T17:59:41.952+0000 7fba6dd8a700 0 -- 172.21.15.87:0/156392408 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x563cf5b7d760 msgr2=0x563cf5b7fb40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:41.969 INFO:tasks.workunit.client.0.smithi087.stderr: Commit image migration: 100% complete...done. 2023-12-24T17:59:41.975 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd namespace create rbd2/ns1 2023-12-24T17:59:42.002 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd namespace create rbd2/ns2 2023-12-24T17:59:42.043 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration prepare rbd2/test1 rbd2/ns1/test1 2023-12-24T17:59:42.129 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:42.132+0000 7fdecfff3700 0 -- 172.21.15.87:0/1720545832 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x55d57005f160 msgr2=0x55d570101a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:42.145 INFO:tasks.workunit.client.0.smithi087.stderr:++ get_migration_state rbd2/ns1/test1 2023-12-24T17:59:42.145 INFO:tasks.workunit.client.0.smithi087.stderr:++ local image=rbd2/ns1/test1 2023-12-24T17:59:42.145 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd --format xml status rbd2/ns1/test1 2023-12-24T17:59:42.145 INFO:tasks.workunit.client.0.smithi087.stderr:++ xmlstarlet sel -t -v //status/migration/state 2023-12-24T17:59:42.204 INFO:tasks.workunit.client.0.smithi087.stderr:+ test prepared = prepared 2023-12-24T17:59:42.205 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration execute rbd2/test1 2023-12-24T17:59:42.267 INFO:tasks.workunit.client.0.smithi087.stderr: Image migration: 3% complete... Image migration: 6% complete... Image migration: 9% complete... Image migration: 12% complete... Image migration: 15% complete... Image migration: 18% complete... Image migration: 21% complete... Image migration: 25% complete... Image migration: 28% complete... Image migration: 31% complete... Image migration: 34% complete... Image migration: 37% complete... Image migration: 40% complete... Image migration: 43% complete... Image migration: 46% complete... Image migration: 50% complete... Image migration: 53% complete... Image migration: 56% complete... Image migration: 59% complete... Image migration: 62% complete... Image migration: 65% complete... Image migration: 68% complete... Image migration: 71% complete... Image migration: 75% complete... Image migration: 78% complete... Image migration: 81% complete... Image migration: 84% complete... Image migration: 87% complete... Image migration: 90% complete... Image migration: 93% complete... Image migration: 96% complete...2023-12-24T17:59:42.270+0000 7f91745f9700 0 -- 172.21.15.87:0/502275576 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x55e0518157e0 msgr2=0x55e051873b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:42.278 INFO:tasks.workunit.client.0.smithi087.stderr: Image migration: 100% complete...done. 2023-12-24T17:59:42.279 INFO:tasks.workunit.client.0.smithi087.stderr:++ get_migration_state rbd2/ns1/test1 2023-12-24T17:59:42.280 INFO:tasks.workunit.client.0.smithi087.stderr:++ local image=rbd2/ns1/test1 2023-12-24T17:59:42.280 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd --format xml status rbd2/ns1/test1 2023-12-24T17:59:42.280 INFO:tasks.workunit.client.0.smithi087.stderr:++ xmlstarlet sel -t -v //status/migration/state 2023-12-24T17:59:42.334 INFO:tasks.workunit.client.0.smithi087.stderr:+ test executed = executed 2023-12-24T17:59:42.334 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration commit rbd2/test1 2023-12-24T17:59:42.412 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:42.409+0000 7f2f9cdad700 0 -- 172.21.15.87:0/1887806529 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x55bd88587850 msgr2=0x7f2f8003f8e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:42.420 INFO:tasks.workunit.client.0.smithi087.stderr: Commit image migration: 3% complete... Commit image migration: 6% complete... Commit image migration: 9% complete... Commit image migration: 12% complete... Commit image migration: 15% complete... Commit image migration: 18% complete... Commit image migration: 21% complete... Commit image migration: 25% complete... Commit image migration: 28% complete... Commit image migration: 31% complete... Commit image migration: 34% complete... Commit image migration: 37% complete... Commit image migration: 40% complete... Commit image migration: 43% complete... Commit image migration: 46% complete... Commit image migration: 50% complete... Commit image migration: 53% complete... Commit image migration: 56% complete... Commit image migration: 59% complete... Commit image migration: 62% complete... Commit image migration: 65% complete... Commit image migration: 68% complete... Commit image migration: 71% complete... Commit image migration: 75% complete... Commit image migration: 78% complete... Commit image migration: 81% complete... Commit image migration: 84% complete... Commit image migration: 87% complete... Commit image migration: 90% complete... Commit image migration: 93% complete... Commit image migration: 96% complete...2023-12-24T17:59:42.422+0000 7f2f9cdad700 0 -- 172.21.15.87:0/1887806529 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x55bd88587850 msgr2=0x7f2f8003f8e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:42.433 INFO:tasks.workunit.client.0.smithi087.stderr: Commit image migration: 100% complete...done. 2023-12-24T17:59:42.435 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration prepare rbd2/ns1/test1 rbd2/ns2/test1 2023-12-24T17:59:42.525 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:42.528+0000 7f6767666700 0 -- 172.21.15.87:0/2151530734 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x55672882f160 msgr2=0x5567288d0970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:42.535 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration execute rbd2/ns2/test1 2023-12-24T17:59:42.601 INFO:tasks.workunit.client.0.smithi087.stderr: Image migration: 3% complete... Image migration: 6% complete... Image migration: 9% complete... Image migration: 12% complete... Image migration: 15% complete... Image migration: 18% complete... Image migration: 21% complete... Image migration: 25% complete... Image migration: 28% complete... Image migration: 31% complete... Image migration: 34% complete... Image migration: 37% complete... Image migration: 40% complete... Image migration: 43% complete... Image migration: 46% complete... Image migration: 50% complete... Image migration: 53% complete... Image migration: 56% complete... Image migration: 59% complete... Image migration: 62% complete... Image migration: 65% complete... Image migration: 68% complete... Image migration: 71% complete... Image migration: 75% complete... Image migration: 78% complete... Image migration: 81% complete... Image migration: 84% complete... Image migration: 87% complete... Image migration: 90% complete... Image migration: 93% complete... Image migration: 96% complete...2023-12-24T17:59:42.604+0000 7f9437bde700 0 -- 172.21.15.87:0/3601271300 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f941c03a8b0 msgr2=0x7f941c03ccd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:42.615 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:42.617+0000 7f9437bde700 0 -- 172.21.15.87:0/3601271300 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x55f83b8de4e0 msgr2=0x7f941c03d360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:42.632 INFO:tasks.workunit.client.0.smithi087.stderr: Image migration: 100% complete...done. 2023-12-24T17:59:42.634 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration commit rbd2/ns2/test1 2023-12-24T17:59:42.711 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:42.713+0000 7f1785f4e700 0 -- 172.21.15.87:0/1975934477 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x5582d310f230 msgr2=0x5582d321ac20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:42.725 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:42.723+0000 7f178674f700 0 -- 172.21.15.87:0/1975934477 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x5582d311f850 msgr2=0x5582d31ca080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:42.739 INFO:tasks.workunit.client.0.smithi087.stderr: Commit image migration: 3% complete... Commit image migration: 6% complete... Commit image migration: 9% complete... Commit image migration: 12% complete... Commit image migration: 15% complete... Commit image migration: 18% complete... Commit image migration: 21% complete... Commit image migration: 25% complete... Commit image migration: 28% complete... Commit image migration: 31% complete... Commit image migration: 34% complete... Commit image migration: 37% complete... Commit image migration: 40% complete... Commit image migration: 43% complete... Commit image migration: 46% complete... Commit image migration: 50% complete... Commit image migration: 53% complete... Commit image migration: 56% complete... Commit image migration: 59% complete... Commit image migration: 62% complete... Commit image migration: 65% complete... Commit image migration: 68% complete... Commit image migration: 71% complete... Commit image migration: 75% complete... Commit image migration: 78% complete... Commit image migration: 81% complete... Commit image migration: 84% complete... Commit image migration: 87% complete... Commit image migration: 90% complete... Commit image migration: 93% complete... Commit image migration: 96% complete... Commit image migration: 100% complete...done. 2023-12-24T17:59:42.741 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create -s 128M test1 2023-12-24T17:59:42.767 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:42.770+0000 7fefa8483540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:42.777 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration prepare test1 --data-pool rbd2 2023-12-24T17:59:42.845 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info test1 2023-12-24T17:59:42.846 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'data_pool: rbd2' 2023-12-24T17:59:42.881 INFO:tasks.workunit.client.0.smithi087.stdout: data_pool: rbd2 2023-12-24T17:59:42.881 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration execute test1 2023-12-24T17:59:42.961 INFO:tasks.workunit.client.0.smithi087.stderr: Image migration: 3% complete... Image migration: 6% complete... Image migration: 9% complete... Image migration: 12% complete... Image migration: 15% complete... Image migration: 18% complete... Image migration: 21% complete... Image migration: 25% complete... Image migration: 28% complete... Image migration: 31% complete... Image migration: 34% complete... Image migration: 37% complete... Image migration: 40% complete... Image migration: 43% complete... Image migration: 46% complete... Image migration: 50% complete... Image migration: 53% complete... Image migration: 56% complete... Image migration: 59% complete... Image migration: 62% complete... Image migration: 65% complete... Image migration: 68% complete... Image migration: 71% complete... Image migration: 75% complete... Image migration: 78% complete... Image migration: 81% complete... Image migration: 84% complete... Image migration: 87% complete... Image migration: 90% complete... Image migration: 93% complete... Image migration: 96% complete...2023-12-24T17:59:42.964+0000 7f74439c4700 0 -- 172.21.15.87:0/1334675797 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x55ff74a9a580 msgr2=0x55ff74b90a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:42.983 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:42.986+0000 7f74439c4700 0 -- 172.21.15.87:0/1334675797 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f742403a8b0 msgr2=0x7f742403ccd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:42.994 INFO:tasks.workunit.client.0.smithi087.stderr: Image migration: 100% complete...done. 2023-12-24T17:59:42.996 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration commit test1 2023-12-24T17:59:43.082 INFO:tasks.workunit.client.0.smithi087.stderr: Commit image migration: 3% complete... Commit image migration: 6% complete... Commit image migration: 9% complete... Commit image migration: 12% complete... Commit image migration: 15% complete... Commit image migration: 18% complete... Commit image migration: 21% complete... Commit image migration: 25% complete... Commit image migration: 28% complete... Commit image migration: 31% complete... Commit image migration: 34% complete... Commit image migration: 37% complete... Commit image migration: 40% complete... Commit image migration: 43% complete... Commit image migration: 46% complete...2023-12-24T17:59:43.083+0000 7f25ea33d700 0 -- 172.21.15.87:0/2927665856 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7f25d0008740 msgr2=0x7f25d000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:43.095 INFO:tasks.workunit.client.0.smithi087.stderr: Commit image migration: 50% complete... Commit image migration: 53% complete... Commit image migration: 56% complete... Commit image migration: 59% complete... Commit image migration: 62% complete... Commit image migration: 65% complete... Commit image migration: 68% complete... Commit image migration: 71% complete... Commit image migration: 75% complete... Commit image migration: 78% complete... Commit image migration: 81% complete... Commit image migration: 84% complete... Commit image migration: 87% complete... Commit image migration: 90% complete... Commit image migration: 93% complete... Commit image migration: 96% complete...2023-12-24T17:59:43.097+0000 7f25eab3e700 0 -- 172.21.15.87:0/2927665856 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x55e31f761850 msgr2=0x55e31f858cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:43.107 INFO:tasks.workunit.client.0.smithi087.stderr: Commit image migration: 100% complete...done. 2023-12-24T17:59:43.110 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration prepare test1 2023-12-24T17:59:43.197 INFO:tasks.workunit.client.0.smithi087.stderr:+ expect_fail rbd trash mv test1 2023-12-24T17:59:43.197 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv test1 2023-12-24T17:59:43.233 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:43.235+0000 7f4ad5863540 -1 librbd::api::Trash: move: cannot move migrating image to trash 2023-12-24T17:59:43.234 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: deferred delete error: (16) Device or resource busy 2023-12-24T17:59:43.237 INFO:tasks.workunit.client.0.smithi087.stderr:+ return 0 2023-12-24T17:59:43.238 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd trash ls -a 2023-12-24T17:59:43.239 INFO:tasks.workunit.client.0.smithi087.stderr:++ cut -d ' ' -f 1 2023-12-24T17:59:43.265 INFO:tasks.workunit.client.0.smithi087.stderr:+ ID=19f149554b73 2023-12-24T17:59:43.266 INFO:tasks.workunit.client.0.smithi087.stderr:+ expect_fail rbd trash rm 19f149554b73 2023-12-24T17:59:43.266 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash rm 19f149554b73 2023-12-24T17:59:43.298 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:43.301+0000 7fc18f3cd700 -1 librbd::image::RefreshRequest: image being migrated 2023-12-24T17:59:43.299 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:43.301+0000 7fc18f3cd700 -1 librbd::image::OpenRequest: failed to refresh image: (30) Read-only file system 2023-12-24T17:59:43.299 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:43.301+0000 7fc18ebcc700 -1 librbd::ImageState: 0x7fc16804da00 failed to open image: (30) Read-only file system 2023-12-24T17:59:43.299 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:43.301+0000 7fc1853c9700 -1 librbd::image::RemoveRequest: 0x7fc168000b80 handle_open_image: error opening image: (30) Read-only file system 2023-12-24T17:59:43.300 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: remove error: (30) Read-only file system 2023-12-24T17:59:43.300 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 0% complete...failed. 2023-12-24T17:59:43.302 INFO:tasks.workunit.client.0.smithi087.stderr:+ return 0 2023-12-24T17:59:43.302 INFO:tasks.workunit.client.0.smithi087.stderr:+ expect_fail rbd trash restore 19f149554b73 2023-12-24T17:59:43.302 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash restore 19f149554b73 2023-12-24T17:59:43.328 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: restore error: 2023-12-24T17:59:43.331+0000 7f66fb06a540 -1 librbd::api::Trash: restore: Current trash source 'migration' does not match expected: user,mirroring,unknown (4) 2023-12-24T17:59:43.328 INFO:tasks.workunit.client.0.smithi087.stderr:(22) Invalid argument 2023-12-24T17:59:43.330 INFO:tasks.workunit.client.0.smithi087.stderr:+ return 0 2023-12-24T17:59:43.330 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration abort test1 2023-12-24T17:59:43.424 INFO:tasks.workunit.client.0.smithi087.stderr: Abort image migration: 1% complete... Abort image migration: 3% complete... Abort image migration: 4% complete... Abort image migration: 6% complete... Abort image migration: 7% complete... Abort image migration: 9% complete... Abort image migration: 10% complete... Abort image migration: 12% complete... Abort image migration: 14% complete... Abort image migration: 15% complete... Abort image migration: 17% complete... Abort image migration: 18% complete... Abort image migration: 20% complete... Abort image migration: 21% complete... Abort image migration: 23% complete... Abort image migration: 25% complete... Abort image migration: 26% complete... Abort image migration: 28% complete... Abort image migration: 29% complete... Abort image migration: 31% complete... Abort image migration: 32% complete... Abort image migration: 34% complete... Abort image migration: 35% complete... Abort image migration: 37% complete... Abort image migration: 39% complete... Abort image migration: 40% complete... Abort image migration: 42% complete... Abort image migration: 43% complete... Abort image migration: 45% complete... Abort image migration: 46% complete... Abort image migration: 48% complete... Abort image migration: 50% complete... Abort image migration: 51% complete... Abort image migration: 53% complete... Abort image migration: 54% complete... Abort image migration: 56% complete... Abort image migration: 57% complete... Abort image migration: 59% complete... Abort image migration: 60% complete... Abort image migration: 62% complete... Abort image migration: 64% complete... Abort image migration: 65% complete... Abort image migration: 67% complete... Abort image migration: 68% complete... Abort image migration: 70% complete... Abort image migration: 71% complete... Abort image migration: 73% complete... Abort image migration: 75% complete... Abort image migration: 76% complete... Abort image migration: 78% complete... Abort image migration: 79% complete... Abort image migration: 81% complete... Abort image migration: 82% complete... Abort image migration: 84% complete... Abort image migration: 85% complete... Abort image migration: 87% complete... Abort image migration: 89% complete... Abort image migration: 90% complete... Abort image migration: 92% complete... Abort image migration: 93% complete... Abort image migration: 95% complete... Abort image migration: 96% complete... Abort image migration: 98% complete...2023-12-24T17:59:43.427+0000 7f22fead8700 0 -- 172.21.15.87:0/440667512 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7f22d8004be0 msgr2=0x7f22d8006fc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:43.454 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:43.457+0000 7f22fead8700 0 -- 172.21.15.87:0/440667512 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7f22e003a8b0 msgr2=0x7f22e003ccd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:43.462 INFO:tasks.workunit.client.0.smithi087.stderr: Abort image migration: 100% complete...done. 2023-12-24T17:59:43.465 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd remove test1 2023-12-24T17:59:43.546 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-24T17:59:43.550 INFO:tasks.workunit.client.0.smithi087.stderr:+ dd if=/dev/urandom bs=1M count=1 2023-12-24T17:59:43.550 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd --image-format 2 import - test1 2023-12-24T17:59:43.599 INFO:tasks.workunit.client.0.smithi087.stderr:1+0 records in 2023-12-24T17:59:43.599 INFO:tasks.workunit.client.0.smithi087.stderr:1+0 records out 2023-12-24T17:59:43.600 INFO:tasks.workunit.client.0.smithi087.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491871 s, 21.3 MB/s 2023-12-24T17:59:43.643 INFO:tasks.workunit.client.0.smithi087.stderr: Importing image: 100% complete...done. 2023-12-24T17:59:43.646 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd export test1 - 2023-12-24T17:59:43.646 INFO:tasks.workunit.client.0.smithi087.stderr:++ md5sum 2023-12-24T17:59:43.881 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 100% complete...done. 2023-12-24T17:59:43.885 INFO:tasks.workunit.client.0.smithi087.stderr:+ md5sum='44d18441c9330202c8c1d593bdccf44a -' 2023-12-24T17:59:43.885 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create test1@snap1 2023-12-24T17:59:44.145 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T17:59:44.152 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap protect test1@snap1 2023-12-24T17:59:44.192 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create test1@snap2 2023-12-24T17:59:45.165 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T17:59:45.176 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd clone test1@snap1 clone_v1 --rbd_default_clone_format=1 2023-12-24T17:59:45.256 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd clone test1@snap2 clone_v2 --rbd_default_clone_format=2 2023-12-24T17:59:45.342 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info clone_v1 2023-12-24T17:59:45.342 INFO:tasks.workunit.client.0.smithi087.stderr:+ fgrep 'parent: rbd/test1@snap1' 2023-12-24T17:59:45.381 INFO:tasks.workunit.client.0.smithi087.stdout: parent: rbd/test1@snap1 2023-12-24T17:59:45.382 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info clone_v2 2023-12-24T17:59:45.382 INFO:tasks.workunit.client.0.smithi087.stderr:+ fgrep 'parent: rbd/test1@snap2' 2023-12-24T17:59:45.424 INFO:tasks.workunit.client.0.smithi087.stdout: parent: rbd/test1@snap2 2023-12-24T17:59:45.425 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info clone_v2 2023-12-24T17:59:45.425 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'op_features: clone-child' 2023-12-24T17:59:45.465 INFO:tasks.workunit.client.0.smithi087.stdout: op_features: clone-child 2023-12-24T17:59:45.465 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd export clone_v1 - 2023-12-24T17:59:45.465 INFO:tasks.workunit.client.0.smithi087.stderr:++ md5sum 2023-12-24T17:59:45.506 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 100% complete...done. 2023-12-24T17:59:45.509 INFO:tasks.workunit.client.0.smithi087.stderr:+ test '44d18441c9330202c8c1d593bdccf44a -' = '44d18441c9330202c8c1d593bdccf44a -' 2023-12-24T17:59:45.509 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd export clone_v2 - 2023-12-24T17:59:45.509 INFO:tasks.workunit.client.0.smithi087.stderr:++ md5sum 2023-12-24T17:59:45.549 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 100% complete...done. 2023-12-24T17:59:45.552 INFO:tasks.workunit.client.0.smithi087.stderr:+ test '44d18441c9330202c8c1d593bdccf44a -' = '44d18441c9330202c8c1d593bdccf44a -' 2023-12-24T17:59:45.552 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd children test1@snap1 2023-12-24T17:59:45.593 INFO:tasks.workunit.client.0.smithi087.stderr:+ test rbd/clone_v1 = rbd/clone_v1 2023-12-24T17:59:45.594 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd children test1@snap2 2023-12-24T17:59:45.629 INFO:tasks.workunit.client.0.smithi087.stderr:+ test rbd/clone_v2 = rbd/clone_v2 2023-12-24T17:59:45.629 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration prepare test1 rbd2/test2 2023-12-24T17:59:45.712 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:45.715+0000 7fb3de384700 0 -- 172.21.15.87:0/164211997 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7fb3c4008740 msgr2=0x7fb3c400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:47.048 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:47.050+0000 7fb3de384700 0 -- 172.21.15.87:0/164211997 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7fb3c003a7d0 msgr2=0x7fb3c003cbd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:47.129 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info clone_v1 2023-12-24T17:59:47.129 INFO:tasks.workunit.client.0.smithi087.stderr:+ fgrep 'parent: rbd2/test2@snap1' 2023-12-24T17:59:47.184 INFO:tasks.workunit.client.0.smithi087.stdout: parent: rbd2/test2@snap1 2023-12-24T17:59:47.185 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info clone_v2 2023-12-24T17:59:47.185 INFO:tasks.workunit.client.0.smithi087.stderr:+ fgrep 'parent: rbd2/test2@snap2' 2023-12-24T17:59:47.239 INFO:tasks.workunit.client.0.smithi087.stdout: parent: rbd2/test2@snap2 2023-12-24T17:59:47.239 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info clone_v2 2023-12-24T17:59:47.240 INFO:tasks.workunit.client.0.smithi087.stderr:+ fgrep 'op_features: clone-child' 2023-12-24T17:59:47.292 INFO:tasks.workunit.client.0.smithi087.stdout: op_features: clone-child 2023-12-24T17:59:47.292 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd children rbd2/test2@snap1 2023-12-24T17:59:47.346 INFO:tasks.workunit.client.0.smithi087.stderr:+ test rbd/clone_v1 = rbd/clone_v1 2023-12-24T17:59:47.347 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd children rbd2/test2@snap2 2023-12-24T17:59:47.396 INFO:tasks.workunit.client.0.smithi087.stderr:+ test rbd/clone_v2 = rbd/clone_v2 2023-12-24T17:59:47.396 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration execute test1 2023-12-24T17:59:47.478 INFO:tasks.workunit.client.0.smithi087.stderr: Image migration: 100% complete...done. 2023-12-24T17:59:47.481 INFO:tasks.workunit.client.0.smithi087.stderr:+ expect_fail rbd migration commit test1 2023-12-24T17:59:47.481 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration commit test1 2023-12-24T17:59:47.545 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: the image has descendants: 2023-12-24T17:59:47.546 INFO:tasks.workunit.client.0.smithi087.stderr: rbd/clone_v1 2023-12-24T17:59:47.546 INFO:tasks.workunit.client.0.smithi087.stderr: rbd/clone_v2 2023-12-24T17:59:47.546 INFO:tasks.workunit.client.0.smithi087.stderr:Warning: in-use, read-only descendant images will not detect the parent update. 2023-12-24T17:59:47.546 INFO:tasks.workunit.client.0.smithi087.stderr:Ensure no descendant images are opened read-only and run again with force flag. 2023-12-24T17:59:47.548 INFO:tasks.workunit.client.0.smithi087.stderr:+ return 0 2023-12-24T17:59:47.548 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration commit test1 --force 2023-12-24T17:59:47.614 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: the image has descendants: 2023-12-24T17:59:47.614 INFO:tasks.workunit.client.0.smithi087.stderr: rbd/clone_v1 2023-12-24T17:59:47.614 INFO:tasks.workunit.client.0.smithi087.stderr: rbd/clone_v2 2023-12-24T17:59:47.614 INFO:tasks.workunit.client.0.smithi087.stderr:Warning: in-use, read-only descendant images will not detect the parent update. 2023-12-24T17:59:47.614 INFO:tasks.workunit.client.0.smithi087.stderr:Proceeding anyway due to force flag set. 2023-12-24T17:59:47.626 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:47.629+0000 7ff359daa700 0 -- 172.21.15.87:0/4081884132 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x560b030bd200 msgr2=0x560b030bf5e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:47.634 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:47.637+0000 7ff359daa700 0 -- 172.21.15.87:0/4081884132 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7ff33c03a7d0 msgr2=0x7ff33c03cbd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:49.196 INFO:tasks.workunit.client.0.smithi087.stderr: Commit image migration: 100% complete...done. 2023-12-24T17:59:49.200 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd export clone_v1 - 2023-12-24T17:59:49.200 INFO:tasks.workunit.client.0.smithi087.stderr:++ md5sum 2023-12-24T17:59:49.245 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 100% complete...done. 2023-12-24T17:59:49.248 INFO:tasks.workunit.client.0.smithi087.stderr:+ test '44d18441c9330202c8c1d593bdccf44a -' = '44d18441c9330202c8c1d593bdccf44a -' 2023-12-24T17:59:49.249 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd export clone_v2 - 2023-12-24T17:59:49.249 INFO:tasks.workunit.client.0.smithi087.stderr:++ md5sum 2023-12-24T17:59:49.291 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 100% complete...done. 2023-12-24T17:59:49.294 INFO:tasks.workunit.client.0.smithi087.stderr:+ test '44d18441c9330202c8c1d593bdccf44a -' = '44d18441c9330202c8c1d593bdccf44a -' 2023-12-24T17:59:49.294 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration prepare rbd2/test2 test1 2023-12-24T17:59:49.400 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:49.403+0000 7fc1171e7700 0 -- 172.21.15.87:0/2617235677 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7fc0f803a850 msgr2=0x7fc0f803cc70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:50.163 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:50.165+0000 7fc1171e7700 0 -- 172.21.15.87:0/2617235677 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7fc0fc008740 msgr2=0x7fc0f803d2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:51.242 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info clone_v1 2023-12-24T17:59:51.242 INFO:tasks.workunit.client.0.smithi087.stderr:+ fgrep 'parent: rbd/test1@snap1' 2023-12-24T17:59:51.296 INFO:tasks.workunit.client.0.smithi087.stdout: parent: rbd/test1@snap1 2023-12-24T17:59:51.296 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info clone_v2 2023-12-24T17:59:51.296 INFO:tasks.workunit.client.0.smithi087.stderr:+ fgrep 'parent: rbd/test1@snap2' 2023-12-24T17:59:51.348 INFO:tasks.workunit.client.0.smithi087.stdout: parent: rbd/test1@snap2 2023-12-24T17:59:51.349 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info clone_v2 2023-12-24T17:59:51.349 INFO:tasks.workunit.client.0.smithi087.stderr:+ fgrep 'op_features: clone-child' 2023-12-24T17:59:51.401 INFO:tasks.workunit.client.0.smithi087.stdout: op_features: clone-child 2023-12-24T17:59:51.401 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd children test1@snap1 2023-12-24T17:59:51.453 INFO:tasks.workunit.client.0.smithi087.stderr:+ test rbd/clone_v1 = rbd/clone_v1 2023-12-24T17:59:51.453 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd children test1@snap2 2023-12-24T17:59:51.502 INFO:tasks.workunit.client.0.smithi087.stderr:+ test rbd/clone_v2 = rbd/clone_v2 2023-12-24T17:59:51.502 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration execute test1 2023-12-24T17:59:51.604 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:51.607+0000 7f6056e83700 0 -- 172.21.15.87:0/622492061 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7f6034008740 msgr2=0x7f603400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:51.607 INFO:tasks.workunit.client.0.smithi087.stderr: Image migration: 100% complete...done. 2023-12-24T17:59:51.609 INFO:tasks.workunit.client.0.smithi087.stderr:+ expect_fail rbd migration commit test1 2023-12-24T17:59:51.609 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration commit test1 2023-12-24T17:59:51.683 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: the image has descendants: 2023-12-24T17:59:51.683 INFO:tasks.workunit.client.0.smithi087.stderr: rbd/clone_v1 2023-12-24T17:59:51.683 INFO:tasks.workunit.client.0.smithi087.stderr: rbd/clone_v2 2023-12-24T17:59:51.683 INFO:tasks.workunit.client.0.smithi087.stderr:Warning: in-use, read-only descendant images will not detect the parent update. 2023-12-24T17:59:51.683 INFO:tasks.workunit.client.0.smithi087.stderr:Ensure no descendant images are opened read-only and run again with force flag. 2023-12-24T17:59:51.685 INFO:tasks.workunit.client.0.smithi087.stderr:+ return 0 2023-12-24T17:59:51.685 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration commit test1 --force 2023-12-24T17:59:51.762 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: the image has descendants: 2023-12-24T17:59:51.762 INFO:tasks.workunit.client.0.smithi087.stderr: rbd/clone_v1 2023-12-24T17:59:51.762 INFO:tasks.workunit.client.0.smithi087.stderr: rbd/clone_v2 2023-12-24T17:59:51.762 INFO:tasks.workunit.client.0.smithi087.stderr:Warning: in-use, read-only descendant images will not detect the parent update. 2023-12-24T17:59:51.763 INFO:tasks.workunit.client.0.smithi087.stderr:Proceeding anyway due to force flag set. 2023-12-24T17:59:51.765 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:51.767+0000 7f99abc18700 0 -- 172.21.15.87:0/1910744227 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7f9990008740 msgr2=0x7f999000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:51.777 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:51.779+0000 7f99abc18700 0 -- 172.21.15.87:0/1910744227 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7f998c03a830 msgr2=0x7f998c03cc50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:53.209 INFO:tasks.workunit.client.0.smithi087.stderr: Commit image migration: 100% complete...done. 2023-12-24T17:59:53.213 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd export clone_v1 - 2023-12-24T17:59:53.213 INFO:tasks.workunit.client.0.smithi087.stderr:++ md5sum 2023-12-24T17:59:53.254 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 100% complete...done. 2023-12-24T17:59:53.258 INFO:tasks.workunit.client.0.smithi087.stderr:+ test '44d18441c9330202c8c1d593bdccf44a -' = '44d18441c9330202c8c1d593bdccf44a -' 2023-12-24T17:59:53.259 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd export clone_v2 - 2023-12-24T17:59:53.259 INFO:tasks.workunit.client.0.smithi087.stderr:++ md5sum 2023-12-24T17:59:53.298 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 100% complete...done. 2023-12-24T17:59:53.301 INFO:tasks.workunit.client.0.smithi087.stderr:+ test '44d18441c9330202c8c1d593bdccf44a -' = '44d18441c9330202c8c1d593bdccf44a -' 2023-12-24T17:59:53.301 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd remove clone_v1 2023-12-24T17:59:53.410 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:53.413+0000 7f2a99199700 0 -- 172.21.15.87:0/1586079345 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x55d43a6beb50 msgr2=0x55d43a765e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:53.413 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:53.416 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd remove clone_v2 2023-12-24T17:59:53.529 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:53.531+0000 7f64227dc700 0 -- 172.21.15.87:0/2263372310 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f6408008740 msgr2=0x7f640800abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:53.546 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:53.548 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap unprotect test1@snap1 2023-12-24T17:59:53.592 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap purge test1 2023-12-24T17:59:55.186 INFO:tasks.workunit.client.0.smithi087.stderr: Removing all snapshots: 50% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2023-12-24T17:59:55.196 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test1 2023-12-24T17:59:55.311 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:55.313+0000 7fc8ad0c6700 0 -- 172.21.15.87:0/1147409935 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x55ee35018980 msgr2=0x55ee350be7f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-24T17:59:55.325 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:55.327 INFO:tasks.workunit.client.0.smithi087.stderr:+ for format in 1 2 2023-12-24T17:59:55.328 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create -s 128M --image-format 1 test2 2023-12-24T17:59:55.353 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: image format 1 is deprecated 2023-12-24T17:59:55.358 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:55.361+0000 7ff475efc540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:55.372 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration prepare test2 --data-pool rbd2 2023-12-24T17:59:55.445 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2023-12-24T17:59:55.487 INFO:tasks.workunit.client.0.smithi087.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2023-12-24T17:59:55.491 INFO:tasks.workunit.client.0.smithi087.stdout: SEC OPS OPS/SEC BYTES/SEC 2023-12-24T17:59:55.491 INFO:tasks.workunit.client.0.smithi087.stdout:elapsed: 0 ops: 1 ops/sec: 250.006 bytes/sec: 250 KiB/s 2023-12-24T17:59:55.495 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration abort test2 2023-12-24T17:59:55.577 INFO:tasks.workunit.client.0.smithi087.stderr: Abort image migration: 1% complete... Abort image migration: 3% complete... Abort image migration: 4% complete... Abort image migration: 6% complete... Abort image migration: 7% complete... Abort image migration: 9% complete... Abort image migration: 10% complete... Abort image migration: 12% complete... Abort image migration: 14% complete... Abort image migration: 15% complete... Abort image migration: 17% complete... Abort image migration: 18% complete... Abort image migration: 20% complete... Abort image migration: 21% complete... Abort image migration: 23% complete... Abort image migration: 25% complete... Abort image migration: 26% complete... Abort image migration: 28% complete... Abort image migration: 29% complete... Abort image migration: 31% complete... Abort image migration: 32% complete... Abort image migration: 34% complete... Abort image migration: 35% complete... Abort image migration: 37% complete... Abort image migration: 39% complete... Abort image migration: 40% complete... Abort image migration: 42% complete... Abort image migration: 43% complete... Abort image migration: 45% complete... Abort image migration: 46% complete... Abort image migration: 48% complete... Abort image migration: 50% complete... Abort image migration: 51% complete... Abort image migration: 53% complete... Abort image migration: 54% complete... Abort image migration: 56% complete... Abort image migration: 57% complete... Abort image migration: 59% complete... Abort image migration: 60% complete... Abort image migration: 62% complete... Abort image migration: 64% complete... Abort image migration: 65% complete... Abort image migration: 67% complete... Abort image migration: 68% complete... Abort image migration: 70% complete... Abort image migration: 71% complete... Abort image migration: 73% complete... Abort image migration: 75% complete... Abort image migration: 76% complete... Abort image migration: 78% complete... Abort image migration: 79% complete... Abort image migration: 81% complete... Abort image migration: 82% complete... Abort image migration: 84% complete... Abort image migration: 85% complete... Abort image migration: 87% complete... Abort image migration: 89% complete... Abort image migration: 90% complete... Abort image migration: 92% complete... Abort image migration: 93% complete... Abort image migration: 95% complete... Abort image migration: 96% complete... Abort image migration: 98% complete...2023-12-24T17:59:55.580+0000 7ff1514b9700 0 -- 172.21.15.87:0/191857307 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7ff12c008740 msgr2=0x7ff12c00abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:55.601 INFO:tasks.workunit.client.0.smithi087.stderr: Abort image migration: 100% complete...done. 2023-12-24T17:59:55.604 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2023-12-24T17:59:55.640 INFO:tasks.workunit.client.0.smithi087.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2023-12-24T17:59:55.642 INFO:tasks.workunit.client.0.smithi087.stdout: SEC OPS OPS/SEC BYTES/SEC 2023-12-24T17:59:55.643 INFO:tasks.workunit.client.0.smithi087.stdout:elapsed: 0 ops: 1 ops/sec: 333.341 bytes/sec: 333 KiB/s 2023-12-24T17:59:55.648 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test2 2023-12-24T17:59:55.706 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-24T17:59:55.708 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create -s 128M --image-format 1 test2 2023-12-24T17:59:55.732 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: image format 1 is deprecated 2023-12-24T17:59:55.738 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:55.740+0000 7f0e0d4f7540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:55.743 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration prepare test2 --data-pool rbd2 2023-12-24T17:59:55.825 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2023-12-24T17:59:55.867 INFO:tasks.workunit.client.0.smithi087.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2023-12-24T17:59:55.872 INFO:tasks.workunit.client.0.smithi087.stdout: SEC OPS OPS/SEC BYTES/SEC 2023-12-24T17:59:55.872 INFO:tasks.workunit.client.0.smithi087.stdout:elapsed: 0 ops: 1 ops/sec: 200.005 bytes/sec: 200 KiB/s 2023-12-24T17:59:55.881 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration execute test2 2023-12-24T17:59:55.972 INFO:tasks.workunit.client.0.smithi087.stderr: Image migration: 3% complete... Image migration: 6% complete... Image migration: 9% complete... Image migration: 12% complete... Image migration: 15% complete... Image migration: 18% complete... Image migration: 21% complete... Image migration: 25% complete... Image migration: 28% complete... Image migration: 31% complete... Image migration: 34% complete... Image migration: 37% complete... Image migration: 40% complete... Image migration: 43% complete... Image migration: 46% complete... Image migration: 50% complete... Image migration: 53% complete... Image migration: 56% complete... Image migration: 59% complete... Image migration: 62% complete... Image migration: 65% complete... Image migration: 68% complete... Image migration: 71% complete... Image migration: 75% complete... Image migration: 78% complete... Image migration: 81% complete... Image migration: 84% complete... Image migration: 87% complete... Image migration: 90% complete... Image migration: 93% complete... Image migration: 96% complete...2023-12-24T17:59:55.975+0000 7f2c04078700 0 -- 172.21.15.87:0/1545999346 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x55b102948b20 msgr2=0x55b102a3ae70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:55.995 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:55.998+0000 7f2c04078700 0 -- 172.21.15.87:0/1545999346 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7f2be403a9d0 msgr2=0x7f2be403cdf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:56.006 INFO:tasks.workunit.client.0.smithi087.stderr: Image migration: 100% complete...done. 2023-12-24T17:59:56.009 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration abort test2 2023-12-24T17:59:56.097 INFO:tasks.workunit.client.0.smithi087.stderr: Abort image migration: 1% complete... Abort image migration: 3% complete... Abort image migration: 4% complete... Abort image migration: 6% complete... Abort image migration: 7% complete... Abort image migration: 9% complete... Abort image migration: 10% complete... Abort image migration: 12% complete... Abort image migration: 14% complete... Abort image migration: 15% complete... Abort image migration: 17% complete... Abort image migration: 18% complete... Abort image migration: 20% complete... Abort image migration: 21% complete... Abort image migration: 23% complete... Abort image migration: 25% complete... Abort image migration: 26% complete... Abort image migration: 28% complete... Abort image migration: 29% complete... Abort image migration: 31% complete... Abort image migration: 32% complete... Abort image migration: 34% complete... Abort image migration: 35% complete... Abort image migration: 37% complete... Abort image migration: 39% complete... Abort image migration: 40% complete... Abort image migration: 42% complete... Abort image migration: 43% complete... Abort image migration: 45% complete... Abort image migration: 46% complete... Abort image migration: 48% complete... Abort image migration: 50% complete... Abort image migration: 51% complete... Abort image migration: 53% complete... Abort image migration: 54% complete... Abort image migration: 56% complete... Abort image migration: 57% complete... Abort image migration: 59% complete... Abort image migration: 60% complete... Abort image migration: 62% complete... Abort image migration: 64% complete... Abort image migration: 65% complete... Abort image migration: 67% complete... Abort image migration: 68% complete... Abort image migration: 70% complete... Abort image migration: 71% complete... Abort image migration: 73% complete... Abort image migration: 75% complete... Abort image migration: 76% complete... Abort image migration: 78% complete... Abort image migration: 79% complete... Abort image migration: 81% complete... Abort image migration: 82% complete... Abort image migration: 84% complete... Abort image migration: 85% complete... Abort image migration: 87% complete... Abort image migration: 89% complete... Abort image migration: 90% complete... Abort image migration: 92% complete... Abort image migration: 93% complete... Abort image migration: 95% complete... Abort image migration: 96% complete... Abort image migration: 98% complete...2023-12-24T17:59:56.100+0000 7f9af3075700 0 -- 172.21.15.87:0/1161141385 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x5597c8fa17e0 msgr2=0x5597c8fffc30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:56.124 INFO:tasks.workunit.client.0.smithi087.stderr: Abort image migration: 100% complete...done. 2023-12-24T17:59:56.126 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2023-12-24T17:59:56.363 INFO:tasks.workunit.client.0.smithi087.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2023-12-24T17:59:56.370 INFO:tasks.workunit.client.0.smithi087.stdout: SEC OPS OPS/SEC BYTES/SEC 2023-12-24T17:59:56.370 INFO:tasks.workunit.client.0.smithi087.stdout:elapsed: 0 ops: 1 ops/sec: 125.003 bytes/sec: 125 KiB/s 2023-12-24T17:59:56.377 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test2 2023-12-24T17:59:56.436 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-24T17:59:56.438 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create -s 128M --image-format 1 test2 2023-12-24T17:59:56.461 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: image format 1 is deprecated 2023-12-24T17:59:56.467 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:56.470+0000 7f0aaa0b2540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:56.473 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration prepare test2 --data-pool INVALID_DATA_POOL 2023-12-24T17:59:56.515 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:56.518+0000 7f3e193d3540 -1 librbd::image::CreateRequest: 0x55e18ccbef40 validate_data_pool: data pool INVALID_DATA_POOL does not exist 2023-12-24T17:59:56.516 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:56.518+0000 7f3e193d3540 -1 librbd::Migration: create_dst_image: header creation failed: (2) No such file or directory 2023-12-24T17:59:56.521 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: preparing migration failed: (2) No such file or directory 2023-12-24T17:59:56.524 INFO:tasks.workunit.client.0.smithi087.stderr:+ true 2023-12-24T17:59:56.524 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2023-12-24T17:59:56.559 INFO:tasks.workunit.client.0.smithi087.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2023-12-24T17:59:56.562 INFO:tasks.workunit.client.0.smithi087.stdout: SEC OPS OPS/SEC BYTES/SEC 2023-12-24T17:59:56.562 INFO:tasks.workunit.client.0.smithi087.stdout:elapsed: 0 ops: 1 ops/sec: 333.341 bytes/sec: 333 KiB/s 2023-12-24T17:59:56.567 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test2 2023-12-24T17:59:56.618 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-24T17:59:56.620 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create -s 128M --image-format 1 test2 2023-12-24T17:59:56.643 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: image format 1 is deprecated 2023-12-24T17:59:56.649 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:56.652+0000 7ff0891e2540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:56.656 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration prepare test2 rbd2/test2 2023-12-24T17:59:56.725 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 rbd2/test2 2023-12-24T17:59:56.765 INFO:tasks.workunit.client.0.smithi087.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2023-12-24T17:59:56.770 INFO:tasks.workunit.client.0.smithi087.stdout: SEC OPS OPS/SEC BYTES/SEC 2023-12-24T17:59:56.770 INFO:tasks.workunit.client.0.smithi087.stdout:elapsed: 0 ops: 1 ops/sec: 200.005 bytes/sec: 200 KiB/s 2023-12-24T17:59:56.776 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration abort test2 2023-12-24T17:59:56.858 INFO:tasks.workunit.client.0.smithi087.stderr: Abort image migration: 1% complete... Abort image migration: 3% complete... Abort image migration: 4% complete... Abort image migration: 6% complete... Abort image migration: 7% complete... Abort image migration: 9% complete... Abort image migration: 10% complete... Abort image migration: 12% complete... Abort image migration: 14% complete... Abort image migration: 15% complete... Abort image migration: 17% complete... Abort image migration: 18% complete... Abort image migration: 20% complete... Abort image migration: 21% complete... Abort image migration: 23% complete... Abort image migration: 25% complete... Abort image migration: 26% complete... Abort image migration: 28% complete... Abort image migration: 29% complete... Abort image migration: 31% complete... Abort image migration: 32% complete... Abort image migration: 34% complete... Abort image migration: 35% complete... Abort image migration: 37% complete... Abort image migration: 39% complete... Abort image migration: 40% complete... Abort image migration: 42% complete... Abort image migration: 43% complete... Abort image migration: 45% complete... Abort image migration: 46% complete... Abort image migration: 48% complete... Abort image migration: 50% complete... Abort image migration: 51% complete... Abort image migration: 53% complete... Abort image migration: 54% complete... Abort image migration: 56% complete... Abort image migration: 57% complete... Abort image migration: 59% complete... Abort image migration: 60% complete... Abort image migration: 62% complete... Abort image migration: 64% complete... Abort image migration: 65% complete... Abort image migration: 67% complete... Abort image migration: 68% complete... Abort image migration: 70% complete... Abort image migration: 71% complete... Abort image migration: 73% complete... Abort image migration: 75% complete... Abort image migration: 76% complete... Abort image migration: 78% complete... Abort image migration: 79% complete... Abort image migration: 81% complete... Abort image migration: 82% complete... Abort image migration: 84% complete... Abort image migration: 85% complete... Abort image migration: 87% complete... Abort image migration: 89% complete... Abort image migration: 90% complete... Abort image migration: 92% complete... Abort image migration: 93% complete... Abort image migration: 95% complete... Abort image migration: 96% complete... Abort image migration: 98% complete... Abort image migration: 100% complete...done. 2023-12-24T17:59:56.860 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2023-12-24T17:59:56.895 INFO:tasks.workunit.client.0.smithi087.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2023-12-24T17:59:56.898 INFO:tasks.workunit.client.0.smithi087.stdout: SEC OPS OPS/SEC BYTES/SEC 2023-12-24T17:59:56.898 INFO:tasks.workunit.client.0.smithi087.stdout:elapsed: 0 ops: 1 ops/sec: 333.341 bytes/sec: 333 KiB/s 2023-12-24T17:59:56.902 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test2 2023-12-24T17:59:56.952 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-24T17:59:56.954 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create -s 128M --image-format 1 test2 2023-12-24T17:59:56.977 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: image format 1 is deprecated 2023-12-24T17:59:56.983 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:56.986+0000 7f9dc2243540 -1 librbd: Forced V1 image creation. 2023-12-24T17:59:56.990 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration prepare test2 rbd2/test2 2023-12-24T17:59:57.064 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration abort rbd2/test2 2023-12-24T17:59:57.135 INFO:tasks.workunit.client.0.smithi087.stderr: Abort image migration: 1% complete... Abort image migration: 3% complete... Abort image migration: 4% complete... Abort image migration: 6% complete... Abort image migration: 7% complete... Abort image migration: 9% complete... Abort image migration: 10% complete... Abort image migration: 12% complete... Abort image migration: 14% complete... Abort image migration: 15% complete... Abort image migration: 17% complete... Abort image migration: 18% complete... Abort image migration: 20% complete... Abort image migration: 21% complete... Abort image migration: 23% complete... Abort image migration: 25% complete... Abort image migration: 26% complete... Abort image migration: 28% complete... Abort image migration: 29% complete... Abort image migration: 31% complete... Abort image migration: 32% complete... Abort image migration: 34% complete... Abort image migration: 35% complete... Abort image migration: 37% complete... Abort image migration: 39% complete... Abort image migration: 40% complete... Abort image migration: 42% complete... Abort image migration: 43% complete... Abort image migration: 45% complete... Abort image migration: 46% complete... Abort image migration: 48% complete... Abort image migration: 50% complete... Abort image migration: 51% complete... Abort image migration: 53% complete... Abort image migration: 54% complete... Abort image migration: 56% complete... Abort image migration: 57% complete... Abort image migration: 59% complete... Abort image migration: 60% complete... Abort image migration: 62% complete... Abort image migration: 64% complete... Abort image migration: 65% complete... Abort image migration: 67% complete... Abort image migration: 68% complete... Abort image migration: 70% complete... Abort image migration: 71% complete... Abort image migration: 73% complete... Abort image migration: 75% complete... Abort image migration: 76% complete... Abort image migration: 78% complete... Abort image migration: 79% complete... Abort image migration: 81% complete... Abort image migration: 82% complete... Abort image migration: 84% complete... Abort image migration: 85% complete... Abort image migration: 87% complete... Abort image migration: 89% complete... Abort image migration: 90% complete... Abort image migration: 92% complete... Abort image migration: 93% complete... Abort image migration: 95% complete... Abort image migration: 96% complete... Abort image migration: 98% complete...2023-12-24T17:59:57.137+0000 7fbd81e0d700 0 -- 172.21.15.87:0/889142257 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x55ca5da9a7e0 msgr2=0x55ca5daf9cb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-24T17:59:57.146 INFO:tasks.workunit.client.0.smithi087.stderr: Abort image migration: 100% complete...done. 2023-12-24T17:59:57.149 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2023-12-24T17:59:57.184 INFO:tasks.workunit.client.0.smithi087.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2023-12-24T17:59:57.187 INFO:tasks.workunit.client.0.smithi087.stdout: SEC OPS OPS/SEC BYTES/SEC 2023-12-24T17:59:57.187 INFO:tasks.workunit.client.0.smithi087.stdout:elapsed: 0 ops: 1 ops/sec: 333.341 bytes/sec: 333 KiB/s 2023-12-24T17:59:57.194 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test2 2023-12-24T17:59:57.255 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-24T17:59:57.258 INFO:tasks.workunit.client.0.smithi087.stderr:+ test 1 = 1 2023-12-24T17:59:57.258 INFO:tasks.workunit.client.0.smithi087.stderr:+ continue 2023-12-24T17:59:57.258 INFO:tasks.workunit.client.0.smithi087.stderr:+ for format in 1 2 2023-12-24T17:59:57.258 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create -s 128M --image-format 2 test2 2023-12-24T17:59:57.307 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration prepare test2 --data-pool rbd2 2023-12-24T17:59:57.407 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:57.409+0000 7f1d29746700 0 -- 172.21.15.87:0/3640837993 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f1d040064f0 msgr2=0x7f1d04012840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:57.412 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2023-12-24T17:59:57.459 INFO:tasks.workunit.client.0.smithi087.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2023-12-24T17:59:57.473 INFO:tasks.workunit.client.0.smithi087.stdout: SEC OPS OPS/SEC BYTES/SEC 2023-12-24T17:59:57.474 INFO:tasks.workunit.client.0.smithi087.stdout:elapsed: 0 ops: 1 ops/sec: 71.4302 bytes/sec: 71 KiB/s 2023-12-24T17:59:57.481 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration abort test2 2023-12-24T17:59:57.603 INFO:tasks.workunit.client.0.smithi087.stderr: Abort image migration: 1% complete... Abort image migration: 3% complete... Abort image migration: 4% complete... Abort image migration: 6% complete... Abort image migration: 7% complete... Abort image migration: 9% complete... Abort image migration: 10% complete... Abort image migration: 12% complete... Abort image migration: 14% complete... Abort image migration: 15% complete... Abort image migration: 17% complete... Abort image migration: 18% complete... Abort image migration: 20% complete... Abort image migration: 21% complete... Abort image migration: 23% complete... Abort image migration: 25% complete... Abort image migration: 26% complete... Abort image migration: 28% complete... Abort image migration: 29% complete... Abort image migration: 31% complete... Abort image migration: 32% complete... Abort image migration: 34% complete... Abort image migration: 35% complete... Abort image migration: 37% complete... Abort image migration: 39% complete... Abort image migration: 40% complete... Abort image migration: 42% complete... Abort image migration: 43% complete... Abort image migration: 45% complete... Abort image migration: 46% complete... Abort image migration: 48% complete... Abort image migration: 50% complete... Abort image migration: 51% complete... Abort image migration: 53% complete... Abort image migration: 54% complete... Abort image migration: 56% complete... Abort image migration: 57% complete... Abort image migration: 59% complete... Abort image migration: 60% complete... Abort image migration: 62% complete... Abort image migration: 64% complete... Abort image migration: 65% complete... Abort image migration: 67% complete... Abort image migration: 68% complete... Abort image migration: 70% complete... Abort image migration: 71% complete... Abort image migration: 73% complete... Abort image migration: 75% complete... Abort image migration: 76% complete... Abort image migration: 78% complete... Abort image migration: 79% complete... Abort image migration: 81% complete... Abort image migration: 82% complete... Abort image migration: 84% complete... Abort image migration: 85% complete... Abort image migration: 87% complete... Abort image migration: 89% complete... Abort image migration: 90% complete... Abort image migration: 92% complete... Abort image migration: 93% complete... Abort image migration: 95% complete... Abort image migration: 96% complete... Abort image migration: 98% complete...2023-12-24T17:59:57.606+0000 7f9770220700 0 -- 172.21.15.87:0/3788392100 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x557a215987e0 msgr2=0x557a215f6ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:57.607 INFO:tasks.workunit.client.0.smithi087.stderr: Abort image migration: 100% complete...done. 2023-12-24T17:59:57.609 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2023-12-24T17:59:57.856 INFO:tasks.workunit.client.0.smithi087.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2023-12-24T17:59:57.859 INFO:tasks.workunit.client.0.smithi087.stdout: SEC OPS OPS/SEC BYTES/SEC 2023-12-24T17:59:57.859 INFO:tasks.workunit.client.0.smithi087.stdout:elapsed: 0 ops: 1 ops/sec: 250.006 bytes/sec: 250 KiB/s 2023-12-24T17:59:57.869 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test2 2023-12-24T17:59:57.961 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete...2023-12-24T17:59:57.964+0000 7f68d3fbc700 0 -- 172.21.15.87:0/1509663504 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x558779cbeb50 msgr2=0x558779d64400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:57.973 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:57.976 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create -s 128M --image-format 2 test2 2023-12-24T17:59:58.018 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration prepare test2 --data-pool rbd2 2023-12-24T17:59:58.133 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:58.136+0000 7fc1bf79d700 0 -- 172.21.15.87:0/1673522561 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x55e116fcefb0 msgr2=0x55e1170bcbf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:58.138 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2023-12-24T17:59:58.187 INFO:tasks.workunit.client.0.smithi087.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2023-12-24T17:59:58.197 INFO:tasks.workunit.client.0.smithi087.stdout: SEC OPS OPS/SEC BYTES/SEC 2023-12-24T17:59:58.198 INFO:tasks.workunit.client.0.smithi087.stdout:elapsed: 0 ops: 1 ops/sec: 90.9111 bytes/sec: 91 KiB/s 2023-12-24T17:59:58.207 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration execute test2 2023-12-24T17:59:58.294 INFO:tasks.workunit.client.0.smithi087.stderr: Image migration: 3% complete... Image migration: 6% complete... Image migration: 9% complete... Image migration: 12% complete... Image migration: 15% complete... Image migration: 18% complete... Image migration: 21% complete... Image migration: 25% complete... Image migration: 28% complete... Image migration: 31% complete... Image migration: 34% complete... Image migration: 37% complete... Image migration: 40% complete... Image migration: 43% complete... Image migration: 46% complete... Image migration: 50% complete... Image migration: 53% complete... Image migration: 56% complete... Image migration: 59% complete... Image migration: 62% complete... Image migration: 65% complete... Image migration: 68% complete... Image migration: 71% complete... Image migration: 75% complete... Image migration: 78% complete... Image migration: 81% complete... Image migration: 84% complete... Image migration: 87% complete... Image migration: 90% complete... Image migration: 93% complete... Image migration: 96% complete...2023-12-24T17:59:58.296+0000 7fd195073700 0 -- 172.21.15.87:0/1259612027 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fd1700064c0 msgr2=0x7fd170006ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:58.317 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:58.320+0000 7fd195874700 0 -- 172.21.15.87:0/1259612027 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x56050c43c980 msgr2=0x56050c531c30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:58.327 INFO:tasks.workunit.client.0.smithi087.stderr: Image migration: 100% complete...done. 2023-12-24T17:59:58.329 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration abort test2 2023-12-24T17:59:58.450 INFO:tasks.workunit.client.0.smithi087.stderr: Abort image migration: 1% complete... Abort image migration: 3% complete... Abort image migration: 4% complete... Abort image migration: 6% complete... Abort image migration: 7% complete... Abort image migration: 9% complete... Abort image migration: 10% complete... Abort image migration: 12% complete... Abort image migration: 14% complete... Abort image migration: 15% complete... Abort image migration: 17% complete... Abort image migration: 18% complete... Abort image migration: 20% complete... Abort image migration: 21% complete... Abort image migration: 23% complete... Abort image migration: 25% complete... Abort image migration: 26% complete... Abort image migration: 28% complete... Abort image migration: 29% complete... Abort image migration: 31% complete... Abort image migration: 32% complete... Abort image migration: 34% complete... Abort image migration: 35% complete... Abort image migration: 37% complete... Abort image migration: 39% complete... Abort image migration: 40% complete... Abort image migration: 42% complete... Abort image migration: 43% complete... Abort image migration: 45% complete... Abort image migration: 46% complete... Abort image migration: 48% complete... Abort image migration: 50% complete... Abort image migration: 51% complete... Abort image migration: 53% complete... Abort image migration: 54% complete... Abort image migration: 56% complete... Abort image migration: 57% complete... Abort image migration: 59% complete... Abort image migration: 60% complete... Abort image migration: 62% complete... Abort image migration: 64% complete... Abort image migration: 65% complete... Abort image migration: 67% complete... Abort image migration: 68% complete... Abort image migration: 70% complete... Abort image migration: 71% complete... Abort image migration: 73% complete... Abort image migration: 75% complete... Abort image migration: 76% complete... Abort image migration: 78% complete... Abort image migration: 79% complete... Abort image migration: 81% complete... Abort image migration: 82% complete... Abort image migration: 84% complete... Abort image migration: 85% complete... Abort image migration: 87% complete... Abort image migration: 89% complete... Abort image migration: 90% complete... Abort image migration: 92% complete... Abort image migration: 93% complete... Abort image migration: 95% complete... Abort image migration: 96% complete... Abort image migration: 98% complete...2023-12-24T17:59:58.453+0000 7fe5334b5700 0 -- 172.21.15.87:0/2972927694 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fe5100087b0 msgr2=0x7fe510007020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:58.452 INFO:tasks.workunit.client.0.smithi087.stderr: Abort image migration: 100% complete...done. 2023-12-24T17:59:58.455 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2023-12-24T17:59:58.497 INFO:tasks.workunit.client.0.smithi087.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2023-12-24T17:59:58.499 INFO:tasks.workunit.client.0.smithi087.stdout: SEC OPS OPS/SEC BYTES/SEC 2023-12-24T17:59:58.499 INFO:tasks.workunit.client.0.smithi087.stdout:elapsed: 0 ops: 1 ops/sec: 1000.02 bytes/sec: 1000 KiB/s 2023-12-24T17:59:58.511 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test2 2023-12-24T17:59:58.645 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete...2023-12-24T17:59:58.647+0000 7f5a1585e700 0 -- 172.21.15.87:0/2428005899 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f59fc008740 msgr2=0x7f59fc00abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:58.654 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:58.657 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create -s 128M --image-format 2 test2 2023-12-24T17:59:58.701 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration prepare test2 --data-pool INVALID_DATA_POOL 2023-12-24T17:59:58.767 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:58.769+0000 7fc20f932540 -1 librbd::image::CreateRequest: 0x55c899e9e420 validate_data_pool: data pool INVALID_DATA_POOL does not exist 2023-12-24T17:59:58.767 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:58.769+0000 7fc20f932540 -1 librbd::Migration: create_dst_image: header creation failed: (2) No such file or directory 2023-12-24T17:59:58.787 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: preparing migration failed: (2) No such file or directory 2023-12-24T17:59:58.789 INFO:tasks.workunit.client.0.smithi087.stderr:+ true 2023-12-24T17:59:58.789 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2023-12-24T17:59:58.833 INFO:tasks.workunit.client.0.smithi087.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2023-12-24T17:59:58.834 INFO:tasks.workunit.client.0.smithi087.stdout: SEC OPS OPS/SEC BYTES/SEC 2023-12-24T17:59:58.834 INFO:tasks.workunit.client.0.smithi087.stdout:elapsed: 0 ops: 1 ops/sec: 250.006 bytes/sec: 250 KiB/s 2023-12-24T17:59:58.845 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test2 2023-12-24T17:59:58.955 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete...2023-12-24T17:59:58.957+0000 7fb5ff774700 0 -- 172.21.15.87:0/1712001105 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x559a2834cb50 msgr2=0x559a283f2860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:58.960 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:58.963 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create -s 128M --image-format 2 test2 2023-12-24T17:59:59.005 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration prepare test2 rbd2/test2 2023-12-24T17:59:59.145 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:59.147+0000 7f588102d700 0 -- 172.21.15.87:0/305729695 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f585c008740 msgr2=0x7f585c00abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:59.150 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 rbd2/test2 2023-12-24T17:59:59.196 INFO:tasks.workunit.client.0.smithi087.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2023-12-24T17:59:59.200 INFO:tasks.workunit.client.0.smithi087.stdout: SEC OPS OPS/SEC BYTES/SEC 2023-12-24T17:59:59.200 INFO:tasks.workunit.client.0.smithi087.stdout:elapsed: 0 ops: 1 ops/sec: 250.006 bytes/sec: 250 KiB/s 2023-12-24T17:59:59.206 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration abort test2 2023-12-24T17:59:59.320 INFO:tasks.workunit.client.0.smithi087.stderr: Abort image migration: 1% complete... Abort image migration: 3% complete... Abort image migration: 4% complete... Abort image migration: 6% complete... Abort image migration: 7% complete... Abort image migration: 9% complete... Abort image migration: 10% complete... Abort image migration: 12% complete... Abort image migration: 14% complete... Abort image migration: 15% complete... Abort image migration: 17% complete... Abort image migration: 18% complete... Abort image migration: 20% complete... Abort image migration: 21% complete... Abort image migration: 23% complete... Abort image migration: 25% complete... Abort image migration: 26% complete... Abort image migration: 28% complete... Abort image migration: 29% complete... Abort image migration: 31% complete... Abort image migration: 32% complete... Abort image migration: 34% complete... Abort image migration: 35% complete... Abort image migration: 37% complete... Abort image migration: 39% complete... Abort image migration: 40% complete... Abort image migration: 42% complete... Abort image migration: 43% complete... Abort image migration: 45% complete... Abort image migration: 46% complete... Abort image migration: 48% complete... Abort image migration: 50% complete... Abort image migration: 51% complete... Abort image migration: 53% complete... Abort image migration: 54% complete... Abort image migration: 56% complete... Abort image migration: 57% complete... Abort image migration: 59% complete... Abort image migration: 60% complete... Abort image migration: 62% complete... Abort image migration: 64% complete... Abort image migration: 65% complete... Abort image migration: 67% complete... Abort image migration: 68% complete... Abort image migration: 70% complete... Abort image migration: 71% complete... Abort image migration: 73% complete... Abort image migration: 75% complete... Abort image migration: 76% complete... Abort image migration: 78% complete... Abort image migration: 79% complete... Abort image migration: 81% complete... Abort image migration: 82% complete... Abort image migration: 84% complete... Abort image migration: 85% complete... Abort image migration: 87% complete... Abort image migration: 89% complete... Abort image migration: 90% complete... Abort image migration: 92% complete... Abort image migration: 93% complete... Abort image migration: 95% complete... Abort image migration: 96% complete... Abort image migration: 98% complete... Abort image migration: 100% complete...done. 2023-12-24T17:59:59.322 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2023-12-24T17:59:59.365 INFO:tasks.workunit.client.0.smithi087.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2023-12-24T17:59:59.369 INFO:tasks.workunit.client.0.smithi087.stdout: SEC OPS OPS/SEC BYTES/SEC 2023-12-24T17:59:59.369 INFO:tasks.workunit.client.0.smithi087.stdout:elapsed: 0 ops: 1 ops/sec: 250.006 bytes/sec: 250 KiB/s 2023-12-24T17:59:59.378 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test2 2023-12-24T17:59:59.482 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete...2023-12-24T17:59:59.486+0000 7f7d7f84d700 0 -- 172.21.15.87:0/2342671787 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x55dd4880cc20 msgr2=0x55dd488c7960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:59.487 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:59.490 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create -s 128M --image-format 2 test2 2023-12-24T17:59:59.536 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration prepare test2 rbd2/test2 2023-12-24T17:59:59.649 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T17:59:59.652+0000 7fac1326f700 0 -- 172.21.15.87:0/3911465631 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x55bc72239160 msgr2=0x55bc72326df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:59.654 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration abort rbd2/test2 2023-12-24T17:59:59.747 INFO:tasks.workunit.client.0.smithi087.stderr: Abort image migration: 1% complete... Abort image migration: 3% complete... Abort image migration: 4% complete... Abort image migration: 6% complete... Abort image migration: 7% complete... Abort image migration: 9% complete... Abort image migration: 10% complete... Abort image migration: 12% complete... Abort image migration: 14% complete... Abort image migration: 15% complete... Abort image migration: 17% complete... Abort image migration: 18% complete... Abort image migration: 20% complete... Abort image migration: 21% complete... Abort image migration: 23% complete... Abort image migration: 25% complete... Abort image migration: 26% complete... Abort image migration: 28% complete... Abort image migration: 29% complete... Abort image migration: 31% complete... Abort image migration: 32% complete... Abort image migration: 34% complete... Abort image migration: 35% complete... Abort image migration: 37% complete... Abort image migration: 39% complete... Abort image migration: 40% complete... Abort image migration: 42% complete... Abort image migration: 43% complete... Abort image migration: 45% complete... Abort image migration: 46% complete... Abort image migration: 48% complete... Abort image migration: 50% complete... Abort image migration: 51% complete... Abort image migration: 53% complete... Abort image migration: 54% complete... Abort image migration: 56% complete... Abort image migration: 57% complete... Abort image migration: 59% complete... Abort image migration: 60% complete... Abort image migration: 62% complete... Abort image migration: 64% complete... Abort image migration: 65% complete... Abort image migration: 67% complete... Abort image migration: 68% complete... Abort image migration: 70% complete... Abort image migration: 71% complete... Abort image migration: 73% complete... Abort image migration: 75% complete... Abort image migration: 76% complete... Abort image migration: 78% complete... Abort image migration: 79% complete... Abort image migration: 81% complete... Abort image migration: 82% complete... Abort image migration: 84% complete... Abort image migration: 85% complete... Abort image migration: 87% complete... Abort image migration: 89% complete... Abort image migration: 90% complete... Abort image migration: 92% complete... Abort image migration: 93% complete... Abort image migration: 95% complete... Abort image migration: 96% complete... Abort image migration: 98% complete... Abort image migration: 100% complete...done. 2023-12-24T17:59:59.749 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2023-12-24T17:59:59.794 INFO:tasks.workunit.client.0.smithi087.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2023-12-24T17:59:59.799 INFO:tasks.workunit.client.0.smithi087.stdout: SEC OPS OPS/SEC BYTES/SEC 2023-12-24T17:59:59.799 INFO:tasks.workunit.client.0.smithi087.stdout:elapsed: 0 ops: 1 ops/sec: 166.67 bytes/sec: 167 KiB/s 2023-12-24T17:59:59.811 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test2 2023-12-24T17:59:59.931 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete...2023-12-24T17:59:59.935+0000 7f915fc8d700 0 -- 172.21.15.87:0/852845258 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f9144008740 msgr2=0x7f914400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T17:59:59.942 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T17:59:59.945 INFO:tasks.workunit.client.0.smithi087.stderr:+ test 2 = 1 2023-12-24T17:59:59.945 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create -s 128M --image-format 2 test2 2023-12-24T17:59:59.988 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration prepare test2 rbd2/ns1/test3 2023-12-24T18:00:00.082 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:00:00.086+0000 7f899890f700 0 -- 172.21.15.87:0/215500150 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x5623248d94e0 msgr2=0x5623249e5810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:00:00.088 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 rbd2/ns1/test3 2023-12-24T18:00:00.136 INFO:tasks.workunit.client.0.smithi087.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2023-12-24T18:00:00.140 INFO:tasks.workunit.client.0.smithi087.stdout: SEC OPS OPS/SEC BYTES/SEC 2023-12-24T18:00:00.140 INFO:tasks.workunit.client.0.smithi087.stdout:elapsed: 0 ops: 1 ops/sec: 250.006 bytes/sec: 250 KiB/s 2023-12-24T18:00:00.145 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd migration abort test2 2023-12-24T18:00:00.249 INFO:tasks.workunit.client.0.smithi087.stderr: Abort image migration: 1% complete... Abort image migration: 3% complete... Abort image migration: 4% complete... Abort image migration: 6% complete... Abort image migration: 7% complete... Abort image migration: 9% complete... Abort image migration: 10% complete... Abort image migration: 12% complete... Abort image migration: 14% complete... Abort image migration: 15% complete... Abort image migration: 17% complete... Abort image migration: 18% complete... Abort image migration: 20% complete... Abort image migration: 21% complete... Abort image migration: 23% complete... Abort image migration: 25% complete... Abort image migration: 26% complete... Abort image migration: 28% complete... Abort image migration: 29% complete... Abort image migration: 31% complete... Abort image migration: 32% complete... Abort image migration: 34% complete... Abort image migration: 35% complete... Abort image migration: 37% complete... Abort image migration: 39% complete... Abort image migration: 40% complete... Abort image migration: 42% complete... Abort image migration: 43% complete... Abort image migration: 45% complete... Abort image migration: 46% complete... Abort image migration: 48% complete... Abort image migration: 50% complete... Abort image migration: 51% complete... Abort image migration: 53% complete... Abort image migration: 54% complete... Abort image migration: 56% complete... Abort image migration: 57% complete... Abort image migration: 59% complete... Abort image migration: 60% complete... Abort image migration: 62% complete... Abort image migration: 64% complete... Abort image migration: 65% complete... Abort image migration: 67% complete... Abort image migration: 68% complete... Abort image migration: 70% complete... Abort image migration: 71% complete... Abort image migration: 73% complete... Abort image migration: 75% complete... Abort image migration: 76% complete... Abort image migration: 78% complete... Abort image migration: 79% complete... Abort image migration: 81% complete... Abort image migration: 82% complete... Abort image migration: 84% complete... Abort image migration: 85% complete... Abort image migration: 87% complete... Abort image migration: 89% complete... Abort image migration: 90% complete... Abort image migration: 92% complete... Abort image migration: 93% complete... Abort image migration: 95% complete... Abort image migration: 96% complete... Abort image migration: 98% complete... Abort image migration: 100% complete...done. 2023-12-24T18:00:00.251 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd bench --io-type write --io-size 1024 --io-total 1024 test2 2023-12-24T18:00:00.297 INFO:tasks.workunit.client.0.smithi087.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2023-12-24T18:00:00.302 INFO:tasks.workunit.client.0.smithi087.stdout: SEC OPS OPS/SEC BYTES/SEC 2023-12-24T18:00:00.302 INFO:tasks.workunit.client.0.smithi087.stdout:elapsed: 0 ops: 1 ops/sec: 200.005 bytes/sec: 200 KiB/s 2023-12-24T18:00:00.310 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test2 2023-12-24T18:00:00.407 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete...2023-12-24T18:00:00.411+0000 7f89cf606700 0 -- 172.21.15.87:0/1594180557 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x559c9a9257c0 msgr2=0x559c9a901a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:00:00.412 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T18:00:00.415 INFO:tasks.workunit.client.0.smithi087.stderr:+ remove_images 2023-12-24T18:00:00.415 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:00.495 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:00.578 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:00.658 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:00.740 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:00.822 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:00.902 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:00.984 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:01.064 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:01.141 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:01.219 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:01.296 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:01.376 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:01.456 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:01.535 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:01.613 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:01.692 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:01.770 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:01.847 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph osd pool rm rbd2 rbd2 --yes-i-really-really-mean-it 2023-12-24T18:00:03.175 INFO:tasks.workunit.client.0.smithi087.stderr:pool 'rbd2' does not exist 2023-12-24T18:00:03.185 INFO:tasks.workunit.client.0.smithi087.stderr:+ test_config 2023-12-24T18:00:03.185 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'testing config...' 2023-12-24T18:00:03.185 INFO:tasks.workunit.client.0.smithi087.stdout:testing config... 2023-12-24T18:00:03.185 INFO:tasks.workunit.client.0.smithi087.stderr:+ remove_images 2023-12-24T18:00:03.185 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:03.272 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:03.353 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:03.432 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:03.508 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:03.584 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:03.658 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:03.933 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:04.008 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:04.083 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:04.165 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:04.241 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:04.321 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:04.401 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:04.480 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:04.555 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:04.633 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:04.709 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:04.785 INFO:tasks.workunit.client.0.smithi087.stderr:+ expect_fail rbd config global set osd rbd_cache true 2023-12-24T18:00:04.785 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd config global set osd rbd_cache true 2023-12-24T18:00:04.808 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: invalid config entity: osd (must be global, client or client.) 2023-12-24T18:00:04.809 INFO:tasks.workunit.client.0.smithi087.stderr:+ return 0 2023-12-24T18:00:04.809 INFO:tasks.workunit.client.0.smithi087.stderr:+ expect_fail rbd config global set global debug_ms 10 2023-12-24T18:00:04.810 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd config global set global debug_ms 10 2023-12-24T18:00:04.832 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: not rbd option: debug_ms 2023-12-24T18:00:04.834 INFO:tasks.workunit.client.0.smithi087.stderr:+ return 0 2023-12-24T18:00:04.834 INFO:tasks.workunit.client.0.smithi087.stderr:+ expect_fail rbd config global set global rbd_UNKNOWN false 2023-12-24T18:00:04.834 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd config global set global rbd_UNKNOWN false 2023-12-24T18:00:04.856 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: invalid config key: rbd_UNKNOWN 2023-12-24T18:00:04.858 INFO:tasks.workunit.client.0.smithi087.stderr:+ return 0 2023-12-24T18:00:04.858 INFO:tasks.workunit.client.0.smithi087.stderr:+ expect_fail rbd config global set global rbd_cache INVALID 2023-12-24T18:00:04.858 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd config global set global rbd_cache INVALID 2023-12-24T18:00:04.885 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: error setting rbd_cache: error parsing value: Expected option value to be integer, got 'INVALID' 2023-12-24T18:00:04.887 INFO:tasks.workunit.client.0.smithi087.stderr:+ return 0 2023-12-24T18:00:04.887 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd config global set global rbd_cache false 2023-12-24T18:00:04.921 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd config global set client rbd_cache true 2023-12-24T18:00:04.957 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd config global set client.123 rbd_cache false 2023-12-24T18:00:04.993 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd config global get global rbd_cache 2023-12-24T18:00:04.993 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^false$' 2023-12-24T18:00:05.021 INFO:tasks.workunit.client.0.smithi087.stdout:false 2023-12-24T18:00:05.021 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd config global get client rbd_cache 2023-12-24T18:00:05.022 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^true$' 2023-12-24T18:00:05.050 INFO:tasks.workunit.client.0.smithi087.stdout:true 2023-12-24T18:00:05.051 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd config global get client.123 rbd_cache 2023-12-24T18:00:05.051 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^false$' 2023-12-24T18:00:05.080 INFO:tasks.workunit.client.0.smithi087.stdout:false 2023-12-24T18:00:05.080 INFO:tasks.workunit.client.0.smithi087.stderr:+ expect_fail rbd config global get client.UNKNOWN rbd_cache 2023-12-24T18:00:05.080 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd config global get client.UNKNOWN rbd_cache 2023-12-24T18:00:05.107 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: rbd_cache is not set 2023-12-24T18:00:05.108 INFO:tasks.workunit.client.0.smithi087.stderr:+ return 0 2023-12-24T18:00:05.109 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd config global list global 2023-12-24T18:00:05.109 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^rbd_cache * false * global *$' 2023-12-24T18:00:05.136 INFO:tasks.workunit.client.0.smithi087.stdout:rbd_cache false global 2023-12-24T18:00:05.137 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd config global list client 2023-12-24T18:00:05.137 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^rbd_cache * true * client *$' 2023-12-24T18:00:05.164 INFO:tasks.workunit.client.0.smithi087.stdout:rbd_cache true client 2023-12-24T18:00:05.165 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd config global list client.123 2023-12-24T18:00:05.165 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^rbd_cache * false * client.123 *$' 2023-12-24T18:00:05.193 INFO:tasks.workunit.client.0.smithi087.stdout:rbd_cache false client.123 2023-12-24T18:00:05.193 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd config global list client.UNKNOWN 2023-12-24T18:00:05.193 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^rbd_cache * true * client *$' 2023-12-24T18:00:05.221 INFO:tasks.workunit.client.0.smithi087.stdout:rbd_cache true client 2023-12-24T18:00:05.221 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd config global rm client rbd_cache 2023-12-24T18:00:05.255 INFO:tasks.workunit.client.0.smithi087.stderr:+ expect_fail rbd config global get client rbd_cache 2023-12-24T18:00:05.255 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd config global get client rbd_cache 2023-12-24T18:00:05.282 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: rbd_cache is not set 2023-12-24T18:00:05.284 INFO:tasks.workunit.client.0.smithi087.stderr:+ return 0 2023-12-24T18:00:05.284 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd config global list client 2023-12-24T18:00:05.284 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^rbd_cache * false * global *$' 2023-12-24T18:00:05.312 INFO:tasks.workunit.client.0.smithi087.stdout:rbd_cache false global 2023-12-24T18:00:05.313 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd config global rm client.123 rbd_cache 2023-12-24T18:00:05.346 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd config global rm global rbd_cache 2023-12-24T18:00:05.381 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd config pool set rbd rbd_cache true 2023-12-24T18:00:05.423 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd config pool list rbd 2023-12-24T18:00:05.423 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^rbd_cache * true * pool *$' 2023-12-24T18:00:05.453 INFO:tasks.workunit.client.0.smithi087.stdout:rbd_cache true pool 2023-12-24T18:00:05.453 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd config pool get rbd rbd_cache 2023-12-24T18:00:05.454 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^true$' 2023-12-24T18:00:05.483 INFO:tasks.workunit.client.0.smithi087.stdout:true 2023-12-24T18:00:05.483 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create -s 1 test1 2023-12-24T18:00:05.510 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:00:05.513+0000 7f1af6c68540 -1 librbd: Forced V1 image creation. 2023-12-24T18:00:05.519 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd config image list rbd/test1 2023-12-24T18:00:05.519 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^rbd_cache * true * pool *$' 2023-12-24T18:00:05.564 INFO:tasks.workunit.client.0.smithi087.stdout:rbd_cache true pool 2023-12-24T18:00:05.564 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd config image set rbd/test1 rbd_cache false 2023-12-24T18:00:05.609 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd config image list rbd/test1 2023-12-24T18:00:05.609 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^rbd_cache * false * image *$' 2023-12-24T18:00:05.648 INFO:tasks.workunit.client.0.smithi087.stdout:rbd_cache false image 2023-12-24T18:00:05.649 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd config image get rbd/test1 rbd_cache 2023-12-24T18:00:05.649 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^false$' 2023-12-24T18:00:05.686 INFO:tasks.workunit.client.0.smithi087.stdout:false 2023-12-24T18:00:05.686 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd config image remove rbd/test1 rbd_cache 2023-12-24T18:00:05.728 INFO:tasks.workunit.client.0.smithi087.stderr:+ expect_fail rbd config image get rbd/test1 rbd_cache 2023-12-24T18:00:05.728 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd config image get rbd/test1 rbd_cache 2023-12-24T18:00:05.760 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: rbd_cache is not set 2023-12-24T18:00:05.766 INFO:tasks.workunit.client.0.smithi087.stderr:+ return 0 2023-12-24T18:00:05.766 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd config image list rbd/test1 2023-12-24T18:00:05.766 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^rbd_cache * true * pool *$' 2023-12-24T18:00:05.805 INFO:tasks.workunit.client.0.smithi087.stdout:rbd_cache true pool 2023-12-24T18:00:05.805 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd config pool remove rbd rbd_cache 2023-12-24T18:00:05.835 INFO:tasks.workunit.client.0.smithi087.stderr:+ expect_fail rbd config pool get rbd rbd_cache 2023-12-24T18:00:05.835 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd config pool get rbd rbd_cache 2023-12-24T18:00:05.862 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: rbd_cache is not set 2023-12-24T18:00:05.864 INFO:tasks.workunit.client.0.smithi087.stderr:+ return 0 2023-12-24T18:00:05.865 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd config pool list rbd 2023-12-24T18:00:05.865 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^rbd_cache * true * config *$' 2023-12-24T18:00:05.894 INFO:tasks.workunit.client.0.smithi087.stdout:rbd_cache true config 2023-12-24T18:00:05.894 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test1 2023-12-24T18:00:05.932 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T18:00:05.935 INFO:tasks.workunit.client.0.smithi087.stderr:+ RBD_CREATE_ARGS= 2023-12-24T18:00:05.935 INFO:tasks.workunit.client.0.smithi087.stderr:+ test_others 2023-12-24T18:00:05.935 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'testing import, export, resize, and snapshots...' 2023-12-24T18:00:05.935 INFO:tasks.workunit.client.0.smithi087.stdout:testing import, export, resize, and snapshots... 2023-12-24T18:00:05.936 INFO:tasks.workunit.client.0.smithi087.stderr:+ TMP_FILES='/tmp/img1 /tmp/img1.new /tmp/img2 /tmp/img2.new /tmp/img3 /tmp/img3.new /tmp/img-diff1.new /tmp/img-diff2.new /tmp/img-diff3.new /tmp/img1.snap1 /tmp/img1.snap1 /tmp/img-diff1.snap1' 2023-12-24T18:00:05.936 INFO:tasks.workunit.client.0.smithi087.stderr:+ remove_images 2023-12-24T18:00:05.936 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:06.010 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:06.085 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:06.163 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:06.240 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:06.318 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:06.399 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:06.476 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:06.556 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:06.627 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:06.702 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:06.777 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:06.856 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:06.932 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:07.006 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:07.084 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:07.159 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:07.236 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:07.314 INFO:tasks.workunit.client.0.smithi087.stderr:+ rm -f /tmp/img1 /tmp/img1.new /tmp/img2 /tmp/img2.new /tmp/img3 /tmp/img3.new /tmp/img-diff1.new /tmp/img-diff2.new /tmp/img-diff3.new /tmp/img1.snap1 /tmp/img1.snap1 /tmp/img-diff1.snap1 2023-12-24T18:00:07.315 INFO:tasks.workunit.client.0.smithi087.stderr:+ dd if=/bin/sh of=/tmp/img1 bs=1k count=1 seek=10 2023-12-24T18:00:07.316 INFO:tasks.workunit.client.0.smithi087.stderr:1+0 records in 2023-12-24T18:00:07.316 INFO:tasks.workunit.client.0.smithi087.stderr:1+0 records out 2023-12-24T18:00:07.316 INFO:tasks.workunit.client.0.smithi087.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 4.1168e-05 s, 24.9 MB/s 2023-12-24T18:00:07.316 INFO:tasks.workunit.client.0.smithi087.stderr:+ dd if=/bin/dd of=/tmp/img1 bs=1k count=10 seek=100 2023-12-24T18:00:07.317 INFO:tasks.workunit.client.0.smithi087.stderr:10+0 records in 2023-12-24T18:00:07.317 INFO:tasks.workunit.client.0.smithi087.stderr:10+0 records out 2023-12-24T18:00:07.317 INFO:tasks.workunit.client.0.smithi087.stderr:10240 bytes (10 kB, 10 KiB) copied, 6.1901e-05 s, 165 MB/s 2023-12-24T18:00:07.317 INFO:tasks.workunit.client.0.smithi087.stderr:+ dd if=/bin/rm of=/tmp/img1 bs=1k count=100 seek=1000 2023-12-24T18:00:07.317 INFO:tasks.workunit.client.0.smithi087.stderr:70+1 records in 2023-12-24T18:00:07.318 INFO:tasks.workunit.client.0.smithi087.stderr:70+1 records out 2023-12-24T18:00:07.318 INFO:tasks.workunit.client.0.smithi087.stderr:72208 bytes (72 kB, 71 KiB) copied, 0.000157151 s, 459 MB/s 2023-12-24T18:00:07.318 INFO:tasks.workunit.client.0.smithi087.stderr:+ dd if=/bin/ls of=/tmp/img1 bs=1k seek=10000 2023-12-24T18:00:07.319 INFO:tasks.workunit.client.0.smithi087.stderr:140+1 records in 2023-12-24T18:00:07.319 INFO:tasks.workunit.client.0.smithi087.stderr:140+1 records out 2023-12-24T18:00:07.319 INFO:tasks.workunit.client.0.smithi087.stderr:143376 bytes (143 kB, 140 KiB) copied, 0.000292364 s, 490 MB/s 2023-12-24T18:00:07.319 INFO:tasks.workunit.client.0.smithi087.stderr:+ dd if=/bin/ln of=/tmp/img1 bs=1k seek=100000 2023-12-24T18:00:07.320 INFO:tasks.workunit.client.0.smithi087.stderr:70+1 records in 2023-12-24T18:00:07.320 INFO:tasks.workunit.client.0.smithi087.stderr:70+1 records out 2023-12-24T18:00:07.320 INFO:tasks.workunit.client.0.smithi087.stderr:72312 bytes (72 kB, 71 KiB) copied, 0.000161878 s, 447 MB/s 2023-12-24T18:00:07.320 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd import /tmp/img1 testimg1 2023-12-24T18:00:07.346 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:00:07.350+0000 7f1bf1975540 -1 librbd: Forced V1 image creation. 2023-12-24T18:00:07.425 INFO:tasks.workunit.client.0.smithi087.stderr: Importing image: 4% complete... Importing image: 8% complete... Importing image: 12% complete... Importing image: 16% complete... Importing image: 20% complete... Importing image: 24% complete... Importing image: 28% complete... Importing image: 32% complete... Importing image: 36% complete... Importing image: 40% complete... Importing image: 45% complete... Importing image: 49% complete... Importing image: 53% complete... Importing image: 57% complete... Importing image: 61% complete... Importing image: 65% complete... Importing image: 69% complete... Importing image: 73% complete... Importing image: 77% complete... Importing image: 81% complete... Importing image: 85% complete... Importing image: 90% complete... Importing image: 94% complete... Importing image: 98% complete... Importing image: 100% complete...done. 2023-12-24T18:00:07.429 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd resize testimg1 --size=256 --allow-shrink 2023-12-24T18:00:07.464 INFO:tasks.workunit.client.0.smithi087.stderr: Resizing image: 100% complete...done. 2023-12-24T18:00:07.468 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd export testimg1 /tmp/img2 2023-12-24T18:00:07.554 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 1% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 73% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 87% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 98% complete... Exporting image: 100% complete...done. 2023-12-24T18:00:07.557 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create testimg1 --snap=snap1 2023-12-24T18:00:08.054 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T18:00:08.058 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd resize testimg1 --size=128 2023-12-24T18:00:08.091 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: shrinking an image is only allowed with the --allow-shrink flag 2023-12-24T18:00:08.097 INFO:tasks.workunit.client.0.smithi087.stderr:+ true 2023-12-24T18:00:08.097 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd resize testimg1 --size=128 --allow-shrink 2023-12-24T18:00:08.144 INFO:tasks.workunit.client.0.smithi087.stderr: Resizing image: 50% complete... Resizing image: 51% complete... Resizing image: 53% complete... Resizing image: 54% complete... Resizing image: 56% complete... Resizing image: 57% complete... Resizing image: 59% complete... Resizing image: 60% complete... Resizing image: 62% complete... Resizing image: 64% complete... Resizing image: 65% complete... Resizing image: 67% complete... Resizing image: 68% complete... Resizing image: 70% complete... Resizing image: 71% complete... Resizing image: 73% complete... Resizing image: 75% complete... Resizing image: 76% complete... Resizing image: 78% complete... Resizing image: 79% complete... Resizing image: 81% complete... Resizing image: 82% complete... Resizing image: 84% complete... Resizing image: 85% complete... Resizing image: 87% complete... Resizing image: 89% complete... Resizing image: 90% complete... Resizing image: 92% complete... Resizing image: 93% complete... Resizing image: 95% complete... Resizing image: 96% complete... Resizing image: 98% complete... Resizing image: 100% complete...done. 2023-12-24T18:00:08.149 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd export testimg1 /tmp/img3 2023-12-24T18:00:08.222 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-24T18:00:08.225 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info testimg1 2023-12-24T18:00:08.226 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'size 128 MiB' 2023-12-24T18:00:08.258 INFO:tasks.workunit.client.0.smithi087.stdout: size 128 MiB in 32 objects 2023-12-24T18:00:08.258 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info --snap=snap1 testimg1 2023-12-24T18:00:08.258 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'size 256 MiB' 2023-12-24T18:00:08.291 INFO:tasks.workunit.client.0.smithi087.stdout: size 256 MiB in 64 objects 2023-12-24T18:00:08.291 INFO:tasks.workunit.client.0.smithi087.stderr:+ rm -rf /tmp/diff-testimg1-1 /tmp/diff-testimg1-2 2023-12-24T18:00:08.292 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd export-diff testimg1 --snap=snap1 /tmp/diff-testimg1-1 2023-12-24T18:00:08.377 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 3% complete... Exporting image: 37% complete... Exporting image: 100% complete...done. 2023-12-24T18:00:08.381 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd export-diff testimg1 --from-snap=snap1 /tmp/diff-testimg1-2 2023-12-24T18:00:08.444 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 100% complete...done. 2023-12-24T18:00:08.451 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --size=1 testimg-diff1 2023-12-24T18:00:08.480 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:00:08.483+0000 7ff1dc18e540 -1 librbd: Forced V1 image creation. 2023-12-24T18:00:08.487 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd import-diff --sparse-size 8K /tmp/diff-testimg1-1 testimg-diff1 2023-12-24T18:00:09.063 INFO:tasks.workunit.client.0.smithi087.stderr: Importing image diff: 22% complete... Importing image diff: 63% complete... Importing image diff: 99% complete... Importing image diff: 100% complete...done. 2023-12-24T18:00:09.068 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd import-diff --sparse-size 8K /tmp/diff-testimg1-2 testimg-diff1 2023-12-24T18:00:09.331 INFO:tasks.workunit.client.0.smithi087.stderr: Importing image diff: 68% complete... Importing image diff: 96% complete... Importing image diff: 100% complete...done. 2023-12-24T18:00:09.336 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info testimg1 2023-12-24T18:00:09.336 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'size 128 MiB' 2023-12-24T18:00:09.369 INFO:tasks.workunit.client.0.smithi087.stdout: size 128 MiB in 32 objects 2023-12-24T18:00:09.370 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info --snap=snap1 testimg1 2023-12-24T18:00:09.370 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'size 256 MiB' 2023-12-24T18:00:09.403 INFO:tasks.workunit.client.0.smithi087.stdout: size 256 MiB in 64 objects 2023-12-24T18:00:09.404 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info testimg-diff1 2023-12-24T18:00:09.404 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'size 128 MiB' 2023-12-24T18:00:09.435 INFO:tasks.workunit.client.0.smithi087.stdout: size 128 MiB in 32 objects 2023-12-24T18:00:09.436 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info --snap=snap1 testimg-diff1 2023-12-24T18:00:09.436 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'size 256 MiB' 2023-12-24T18:00:09.468 INFO:tasks.workunit.client.0.smithi087.stdout: size 256 MiB in 64 objects 2023-12-24T18:00:09.468 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd copy testimg1 --snap=snap1 testimg2 2023-12-24T18:00:09.558 INFO:tasks.workunit.client.0.smithi087.stderr: Image copy: 1% complete... Image copy: 3% complete... Image copy: 4% complete... Image copy: 6% complete... Image copy: 7% complete... Image copy: 9% complete... Image copy: 10% complete... Image copy: 12% complete... Image copy: 14% complete... Image copy: 15% complete... Image copy: 17% complete... Image copy: 18% complete... Image copy: 20% complete... Image copy: 21% complete... Image copy: 23% complete... Image copy: 25% complete... Image copy: 26% complete... Image copy: 28% complete... Image copy: 29% complete... Image copy: 31% complete... Image copy: 32% complete... Image copy: 34% complete... Image copy: 35% complete... Image copy: 37% complete... Image copy: 39% complete... Image copy: 40% complete... Image copy: 42% complete... Image copy: 43% complete... Image copy: 45% complete... Image copy: 46% complete... Image copy: 48% complete... Image copy: 50% complete... Image copy: 51% complete... Image copy: 53% complete... Image copy: 54% complete... Image copy: 56% complete... Image copy: 57% complete... Image copy: 59% complete... Image copy: 60% complete... Image copy: 62% complete... Image copy: 64% complete... Image copy: 65% complete... Image copy: 67% complete... Image copy: 68% complete... Image copy: 70% complete... Image copy: 71% complete... Image copy: 73% complete... Image copy: 75% complete... Image copy: 76% complete... Image copy: 78% complete... Image copy: 79% complete... Image copy: 81% complete... Image copy: 82% complete... Image copy: 84% complete... Image copy: 85% complete... Image copy: 87% complete... Image copy: 89% complete... Image copy: 90% complete... Image copy: 92% complete... Image copy: 93% complete... Image copy: 95% complete... Image copy: 96% complete... Image copy: 98% complete... Image copy: 100% complete... Image copy: 100% complete...done. 2023-12-24T18:00:09.562 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd copy testimg1 testimg3 2023-12-24T18:00:09.643 INFO:tasks.workunit.client.0.smithi087.stderr: Image copy: 3% complete... Image copy: 6% complete... Image copy: 9% complete... Image copy: 12% complete... Image copy: 15% complete... Image copy: 18% complete... Image copy: 21% complete... Image copy: 25% complete... Image copy: 28% complete... Image copy: 31% complete... Image copy: 34% complete... Image copy: 37% complete... Image copy: 40% complete... Image copy: 43% complete... Image copy: 46% complete... Image copy: 50% complete... Image copy: 53% complete... Image copy: 56% complete... Image copy: 59% complete... Image copy: 62% complete... Image copy: 65% complete... Image copy: 68% complete... Image copy: 71% complete... Image copy: 75% complete... Image copy: 78% complete... Image copy: 81% complete... Image copy: 84% complete... Image copy: 87% complete... Image copy: 90% complete... Image copy: 93% complete... Image copy: 96% complete... Image copy: 100% complete... Image copy: 100% complete...done. 2023-12-24T18:00:09.646 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd copy testimg-diff1 --sparse-size 768K --snap=snap1 testimg-diff2 2023-12-24T18:00:09.739 INFO:tasks.workunit.client.0.smithi087.stderr: Image copy: 1% complete... Image copy: 3% complete... Image copy: 4% complete... Image copy: 6% complete... Image copy: 7% complete... Image copy: 9% complete... Image copy: 10% complete... Image copy: 12% complete... Image copy: 14% complete... Image copy: 15% complete... Image copy: 17% complete... Image copy: 18% complete... Image copy: 20% complete... Image copy: 21% complete... Image copy: 23% complete... Image copy: 25% complete... Image copy: 26% complete... Image copy: 28% complete... Image copy: 29% complete... Image copy: 31% complete... Image copy: 32% complete... Image copy: 34% complete... Image copy: 35% complete... Image copy: 37% complete... Image copy: 39% complete... Image copy: 40% complete... Image copy: 42% complete... Image copy: 43% complete... Image copy: 45% complete... Image copy: 46% complete... Image copy: 48% complete... Image copy: 50% complete... Image copy: 51% complete... Image copy: 53% complete... Image copy: 54% complete... Image copy: 56% complete... Image copy: 57% complete... Image copy: 59% complete... Image copy: 60% complete... Image copy: 62% complete... Image copy: 64% complete... Image copy: 65% complete... Image copy: 67% complete... Image copy: 68% complete... Image copy: 70% complete... Image copy: 71% complete... Image copy: 73% complete... Image copy: 75% complete... Image copy: 76% complete... Image copy: 78% complete... Image copy: 79% complete... Image copy: 81% complete... Image copy: 82% complete... Image copy: 84% complete... Image copy: 85% complete... Image copy: 87% complete... Image copy: 89% complete... Image copy: 90% complete... Image copy: 92% complete... Image copy: 93% complete... Image copy: 95% complete... Image copy: 96% complete... Image copy: 98% complete... Image copy: 100% complete... Image copy: 100% complete...done. 2023-12-24T18:00:09.743 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd copy testimg-diff1 --sparse-size 768K testimg-diff3 2023-12-24T18:00:09.830 INFO:tasks.workunit.client.0.smithi087.stderr: Image copy: 3% complete... Image copy: 6% complete... Image copy: 9% complete... Image copy: 12% complete... Image copy: 15% complete... Image copy: 18% complete... Image copy: 21% complete... Image copy: 25% complete... Image copy: 28% complete... Image copy: 31% complete... Image copy: 34% complete... Image copy: 37% complete... Image copy: 40% complete... Image copy: 43% complete... Image copy: 46% complete... Image copy: 50% complete... Image copy: 53% complete... Image copy: 56% complete... Image copy: 59% complete... Image copy: 62% complete... Image copy: 65% complete... Image copy: 68% complete... Image copy: 71% complete... Image copy: 75% complete... Image copy: 78% complete... Image copy: 81% complete... Image copy: 84% complete... Image copy: 87% complete... Image copy: 90% complete... Image copy: 93% complete... Image copy: 96% complete... Image copy: 100% complete... Image copy: 100% complete...done. 2023-12-24T18:00:09.834 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info testimg2 2023-12-24T18:00:09.834 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'size 256 MiB' 2023-12-24T18:00:09.867 INFO:tasks.workunit.client.0.smithi087.stdout: size 256 MiB in 64 objects 2023-12-24T18:00:09.867 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info testimg3 2023-12-24T18:00:09.867 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'size 128 MiB' 2023-12-24T18:00:09.901 INFO:tasks.workunit.client.0.smithi087.stdout: size 128 MiB in 32 objects 2023-12-24T18:00:09.901 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info testimg-diff2 2023-12-24T18:00:09.901 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'size 256 MiB' 2023-12-24T18:00:09.934 INFO:tasks.workunit.client.0.smithi087.stdout: size 256 MiB in 64 objects 2023-12-24T18:00:09.935 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info testimg-diff3 2023-12-24T18:00:09.935 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'size 128 MiB' 2023-12-24T18:00:09.968 INFO:tasks.workunit.client.0.smithi087.stdout: size 128 MiB in 32 objects 2023-12-24T18:00:09.968 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd deep copy testimg1 testimg4 2023-12-24T18:00:10.125 INFO:tasks.workunit.client.0.smithi087.stderr: Image deep copy: 1% complete... Image deep copy: 3% complete... Image deep copy: 4% complete... Image deep copy: 6% complete... Image deep copy: 7% complete... Image deep copy: 9% complete... Image deep copy: 10% complete... Image deep copy: 12% complete... Image deep copy: 14% complete... Image deep copy: 15% complete... Image deep copy: 17% complete... Image deep copy: 18% complete... Image deep copy: 20% complete... Image deep copy: 21% complete... Image deep copy: 23% complete... Image deep copy: 25% complete... Image deep copy: 26% complete... Image deep copy: 28% complete... Image deep copy: 29% complete... Image deep copy: 31% complete... Image deep copy: 32% complete... Image deep copy: 34% complete... Image deep copy: 35% complete... Image deep copy: 37% complete... Image deep copy: 39% complete... Image deep copy: 40% complete... Image deep copy: 42% complete... Image deep copy: 43% complete... Image deep copy: 45% complete... Image deep copy: 46% complete... Image deep copy: 48% complete... Image deep copy: 50% complete... Image deep copy: 51% complete... Image deep copy: 53% complete... Image deep copy: 54% complete... Image deep copy: 56% complete... Image deep copy: 57% complete... Image deep copy: 59% complete... Image deep copy: 60% complete... Image deep copy: 62% complete... Image deep copy: 64% complete... Image deep copy: 65% complete... Image deep copy: 67% complete... Image deep copy: 68% complete... Image deep copy: 70% complete... Image deep copy: 71% complete... Image deep copy: 73% complete... Image deep copy: 75% complete... Image deep copy: 76% complete... Image deep copy: 78% complete... Image deep copy: 79% complete... Image deep copy: 81% complete... Image deep copy: 82% complete... Image deep copy: 84% complete... Image deep copy: 85% complete... Image deep copy: 87% complete... Image deep copy: 89% complete... Image deep copy: 90% complete... Image deep copy: 92% complete... Image deep copy: 93% complete... Image deep copy: 95% complete... Image deep copy: 96% complete... Image deep copy: 98% complete... Image deep copy: 100% complete... Image deep copy: 100% complete...done. 2023-12-24T18:00:10.128 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd deep copy testimg1 --snap=snap1 testimg5 2023-12-24T18:00:11.134 INFO:tasks.workunit.client.0.smithi087.stderr: Image deep copy: 1% complete... Image deep copy: 3% complete... Image deep copy: 4% complete... Image deep copy: 6% complete... Image deep copy: 7% complete... Image deep copy: 9% complete... Image deep copy: 10% complete... Image deep copy: 12% complete... Image deep copy: 14% complete... Image deep copy: 15% complete... Image deep copy: 17% complete... Image deep copy: 18% complete... Image deep copy: 20% complete... Image deep copy: 21% complete... Image deep copy: 23% complete... Image deep copy: 25% complete... Image deep copy: 26% complete... Image deep copy: 28% complete... Image deep copy: 29% complete... Image deep copy: 31% complete... Image deep copy: 32% complete... Image deep copy: 34% complete... Image deep copy: 35% complete... Image deep copy: 37% complete... Image deep copy: 39% complete... Image deep copy: 40% complete... Image deep copy: 42% complete... Image deep copy: 43% complete... Image deep copy: 45% complete... Image deep copy: 46% complete... Image deep copy: 48% complete... Image deep copy: 50% complete... Image deep copy: 51% complete... Image deep copy: 53% complete... Image deep copy: 54% complete... Image deep copy: 56% complete... Image deep copy: 57% complete... Image deep copy: 59% complete... Image deep copy: 60% complete... Image deep copy: 62% complete... Image deep copy: 64% complete... Image deep copy: 65% complete... Image deep copy: 67% complete... Image deep copy: 68% complete... Image deep copy: 70% complete... Image deep copy: 71% complete... Image deep copy: 73% complete... Image deep copy: 75% complete... Image deep copy: 76% complete... Image deep copy: 78% complete... Image deep copy: 79% complete... Image deep copy: 81% complete... Image deep copy: 82% complete... Image deep copy: 84% complete... Image deep copy: 85% complete... Image deep copy: 87% complete... Image deep copy: 89% complete... Image deep copy: 90% complete... Image deep copy: 92% complete... Image deep copy: 93% complete... Image deep copy: 95% complete... Image deep copy: 96% complete... Image deep copy: 98% complete... Image deep copy: 100% complete... Image deep copy: 100% complete...done. 2023-12-24T18:00:11.138 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info testimg4 2023-12-24T18:00:11.139 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'size 128 MiB' 2023-12-24T18:00:11.174 INFO:tasks.workunit.client.0.smithi087.stdout: size 128 MiB in 32 objects 2023-12-24T18:00:11.175 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info testimg5 2023-12-24T18:00:11.175 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'size 256 MiB' 2023-12-24T18:00:11.208 INFO:tasks.workunit.client.0.smithi087.stdout: size 256 MiB in 64 objects 2023-12-24T18:00:11.209 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap ls testimg4 2023-12-24T18:00:11.209 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -v SNAPID 2023-12-24T18:00:11.209 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:00:11.209 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 1 2023-12-24T18:00:11.242 INFO:tasks.workunit.client.0.smithi087.stdout:1 2023-12-24T18:00:11.243 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap ls testimg4 2023-12-24T18:00:11.243 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '.*snap1.*' 2023-12-24T18:00:11.277 INFO:tasks.workunit.client.0.smithi087.stdout: 18 snap1 256 MiB Sun Dec 24 18:00:10 2023 2023-12-24T18:00:11.277 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd export testimg1 /tmp/img1.new 2023-12-24T18:00:11.345 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-24T18:00:11.348 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd export testimg2 /tmp/img2.new 2023-12-24T18:00:11.450 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 1% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 73% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 87% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 98% complete... Exporting image: 100% complete...done. 2023-12-24T18:00:11.453 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd export testimg3 /tmp/img3.new 2023-12-24T18:00:11.518 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-24T18:00:11.521 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd export testimg-diff1 /tmp/img-diff1.new 2023-12-24T18:00:11.591 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-24T18:00:11.594 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd export testimg-diff2 /tmp/img-diff2.new 2023-12-24T18:00:11.701 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 1% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 73% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 87% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 98% complete... Exporting image: 100% complete...done. 2023-12-24T18:00:11.704 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd export testimg-diff3 /tmp/img-diff3.new 2023-12-24T18:00:11.773 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-24T18:00:11.776 INFO:tasks.workunit.client.0.smithi087.stderr:+ cmp /tmp/img2 /tmp/img2.new 2023-12-24T18:00:11.969 INFO:tasks.workunit.client.0.smithi087.stderr:+ cmp /tmp/img3 /tmp/img3.new 2023-12-24T18:00:12.064 INFO:tasks.workunit.client.0.smithi087.stderr:+ cmp /tmp/img2 /tmp/img-diff2.new 2023-12-24T18:00:12.223 INFO:tasks.workunit.client.0.smithi087.stderr:+ cmp /tmp/img3 /tmp/img-diff3.new 2023-12-24T18:00:12.302 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap rollback --snap=snap1 testimg1 2023-12-24T18:00:12.400 INFO:tasks.workunit.client.0.smithi087.stderr: Rolling back to snapshot: 1% complete... Rolling back to snapshot: 3% complete... Rolling back to snapshot: 4% complete... Rolling back to snapshot: 6% complete... Rolling back to snapshot: 7% complete... Rolling back to snapshot: 9% complete... Rolling back to snapshot: 10% complete... Rolling back to snapshot: 12% complete... Rolling back to snapshot: 14% complete... Rolling back to snapshot: 15% complete... Rolling back to snapshot: 17% complete... Rolling back to snapshot: 18% complete... Rolling back to snapshot: 20% complete... Rolling back to snapshot: 21% complete... Rolling back to snapshot: 23% complete... Rolling back to snapshot: 25% complete... Rolling back to snapshot: 26% complete... Rolling back to snapshot: 28% complete... Rolling back to snapshot: 29% complete... Rolling back to snapshot: 31% complete... Rolling back to snapshot: 32% complete... Rolling back to snapshot: 34% complete... Rolling back to snapshot: 35% complete... Rolling back to snapshot: 37% complete... Rolling back to snapshot: 39% complete... Rolling back to snapshot: 40% complete... Rolling back to snapshot: 42% complete... Rolling back to snapshot: 43% complete... Rolling back to snapshot: 45% complete... Rolling back to snapshot: 46% complete... Rolling back to snapshot: 48% complete... Rolling back to snapshot: 50% complete... Rolling back to snapshot: 51% complete... Rolling back to snapshot: 53% complete... Rolling back to snapshot: 54% complete... Rolling back to snapshot: 56% complete... Rolling back to snapshot: 57% complete... Rolling back to snapshot: 59% complete... Rolling back to snapshot: 60% complete... Rolling back to snapshot: 62% complete... Rolling back to snapshot: 64% complete... Rolling back to snapshot: 65% complete... Rolling back to snapshot: 67% complete... Rolling back to snapshot: 68% complete... Rolling back to snapshot: 70% complete... Rolling back to snapshot: 71% complete... Rolling back to snapshot: 73% complete... Rolling back to snapshot: 75% complete... Rolling back to snapshot: 76% complete... Rolling back to snapshot: 78% complete... Rolling back to snapshot: 79% complete... Rolling back to snapshot: 81% complete... Rolling back to snapshot: 82% complete... Rolling back to snapshot: 84% complete... Rolling back to snapshot: 85% complete... Rolling back to snapshot: 87% complete... Rolling back to snapshot: 89% complete... Rolling back to snapshot: 90% complete... Rolling back to snapshot: 92% complete... Rolling back to snapshot: 93% complete... Rolling back to snapshot: 95% complete... Rolling back to snapshot: 96% complete... Rolling back to snapshot: 98% complete... Rolling back to snapshot: 100% complete...done. 2023-12-24T18:00:12.405 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap rollback --snap=snap1 testimg-diff1 2023-12-24T18:00:12.500 INFO:tasks.workunit.client.0.smithi087.stderr: Rolling back to snapshot: 1% complete... Rolling back to snapshot: 3% complete... Rolling back to snapshot: 4% complete... Rolling back to snapshot: 6% complete... Rolling back to snapshot: 7% complete... Rolling back to snapshot: 9% complete... Rolling back to snapshot: 10% complete... Rolling back to snapshot: 12% complete... Rolling back to snapshot: 14% complete... Rolling back to snapshot: 15% complete... Rolling back to snapshot: 17% complete... Rolling back to snapshot: 18% complete... Rolling back to snapshot: 20% complete... Rolling back to snapshot: 21% complete... Rolling back to snapshot: 23% complete... Rolling back to snapshot: 25% complete... Rolling back to snapshot: 26% complete... Rolling back to snapshot: 28% complete... Rolling back to snapshot: 29% complete... Rolling back to snapshot: 31% complete... Rolling back to snapshot: 32% complete... Rolling back to snapshot: 34% complete... Rolling back to snapshot: 35% complete... Rolling back to snapshot: 37% complete... Rolling back to snapshot: 39% complete... Rolling back to snapshot: 40% complete... Rolling back to snapshot: 42% complete... Rolling back to snapshot: 43% complete... Rolling back to snapshot: 45% complete... Rolling back to snapshot: 46% complete... Rolling back to snapshot: 48% complete... Rolling back to snapshot: 50% complete... Rolling back to snapshot: 51% complete... Rolling back to snapshot: 53% complete... Rolling back to snapshot: 54% complete... Rolling back to snapshot: 56% complete... Rolling back to snapshot: 57% complete... Rolling back to snapshot: 59% complete... Rolling back to snapshot: 60% complete... Rolling back to snapshot: 62% complete... Rolling back to snapshot: 64% complete... Rolling back to snapshot: 65% complete... Rolling back to snapshot: 67% complete... Rolling back to snapshot: 68% complete... Rolling back to snapshot: 70% complete... Rolling back to snapshot: 71% complete... Rolling back to snapshot: 73% complete... Rolling back to snapshot: 75% complete... Rolling back to snapshot: 76% complete... Rolling back to snapshot: 78% complete... Rolling back to snapshot: 79% complete... Rolling back to snapshot: 81% complete... Rolling back to snapshot: 82% complete... Rolling back to snapshot: 84% complete... Rolling back to snapshot: 85% complete... Rolling back to snapshot: 87% complete... Rolling back to snapshot: 89% complete... Rolling back to snapshot: 90% complete... Rolling back to snapshot: 92% complete... Rolling back to snapshot: 93% complete... Rolling back to snapshot: 95% complete... Rolling back to snapshot: 96% complete... Rolling back to snapshot: 98% complete... Rolling back to snapshot: 100% complete...done. 2023-12-24T18:00:12.510 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info testimg1 2023-12-24T18:00:12.510 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'size 256 MiB' 2023-12-24T18:00:12.545 INFO:tasks.workunit.client.0.smithi087.stdout: size 256 MiB in 64 objects 2023-12-24T18:00:12.546 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info testimg-diff1 2023-12-24T18:00:12.546 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'size 256 MiB' 2023-12-24T18:00:12.577 INFO:tasks.workunit.client.0.smithi087.stdout: size 256 MiB in 64 objects 2023-12-24T18:00:12.577 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd export testimg1 /tmp/img1.snap1 2023-12-24T18:00:12.679 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 1% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 73% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 87% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 98% complete... Exporting image: 100% complete...done. 2023-12-24T18:00:12.682 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd export testimg-diff1 /tmp/img-diff1.snap1 2023-12-24T18:00:12.784 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 1% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 73% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 87% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 98% complete... Exporting image: 100% complete...done. 2023-12-24T18:00:12.788 INFO:tasks.workunit.client.0.smithi087.stderr:+ cmp /tmp/img2 /tmp/img1.snap1 2023-12-24T18:00:12.936 INFO:tasks.workunit.client.0.smithi087.stderr:+ cmp /tmp/img2 /tmp/img-diff1.snap1 2023-12-24T18:00:13.084 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm testimg2 2023-12-24T18:00:13.168 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 1% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 12% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 23% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 37% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 48% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 62% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 73% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 87% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 922023-12-24T18:00:13.171+0000 7f658e601700 0 -- 172.21.15.87:0/1059883190 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x5631480f4b50 msgr2=0x56314819ace0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:00:13.188 INFO:tasks.workunit.client.0.smithi087.stderr:% complete... Removing image: 93% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 98% complete...2023-12-24T18:00:13.191+0000 7f658e601700 0 -- 172.21.15.87:0/1059883190 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7f657003a130 msgr2=0x7f657003c510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:00:13.191 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T18:00:13.193 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm testimg3 2023-12-24T18:00:13.297 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-24T18:00:13.299 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create testimg2 -s 0 2023-12-24T18:00:13.327 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:00:13.330+0000 7f5d8a798540 -1 librbd: Forced V1 image creation. 2023-12-24T18:00:13.335 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd cp testimg2 testimg3 2023-12-24T18:00:13.388 INFO:tasks.workunit.client.0.smithi087.stderr: Image copy: 100% complete...done. 2023-12-24T18:00:13.391 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd deep cp testimg2 testimg6 2023-12-24T18:00:13.443 INFO:tasks.workunit.client.0.smithi087.stderr: Image deep copy: 100% complete...done. 2023-12-24T18:00:13.445 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap rm --snap=snap1 testimg1 2023-12-24T18:00:14.075 INFO:tasks.workunit.client.0.smithi087.stderr: Removing snap: 100% complete...done. 2023-12-24T18:00:14.081 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap rm --snap=snap1 testimg-diff1 2023-12-24T18:00:15.093 INFO:tasks.workunit.client.0.smithi087.stderr: Removing snap: 100% complete...done. 2023-12-24T18:00:15.101 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info --snap=snap1 testimg1 2023-12-24T18:00:15.101 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'error setting snapshot context: (2) No such file or directory' 2023-12-24T18:00:15.136 INFO:tasks.workunit.client.0.smithi087.stdout:error setting snapshot context: (2) No such file or directory 2023-12-24T18:00:15.137 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info --snap=snap1 testimg-diff1 2023-12-24T18:00:15.137 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'error setting snapshot context: (2) No such file or directory' 2023-12-24T18:00:15.170 INFO:tasks.workunit.client.0.smithi087.stdout:error setting snapshot context: (2) No such file or directory 2023-12-24T18:00:15.171 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd sparsify testimg1 2023-12-24T18:00:15.431 INFO:tasks.workunit.client.0.smithi087.stderr: Image sparsify: 1% complete... Image sparsify: 3% complete... Image sparsify: 4% complete... Image sparsify: 6% complete... Image sparsify: 7% complete... Image sparsify: 9% complete... Image sparsify: 10% complete... Image sparsify: 12% complete... Image sparsify: 14% complete... Image sparsify: 15% complete... Image sparsify: 17% complete... Image sparsify: 18% complete... Image sparsify: 20% complete... Image sparsify: 21% complete... Image sparsify: 23% complete... Image sparsify: 25% complete... Image sparsify: 26% complete... Image sparsify: 28% complete... Image sparsify: 29% complete... Image sparsify: 31% complete... Image sparsify: 32% complete... Image sparsify: 34% complete... Image sparsify: 35% complete... Image sparsify: 37% complete... Image sparsify: 39% complete... Image sparsify: 40% complete... Image sparsify: 42% complete... Image sparsify: 43% complete... Image sparsify: 45% complete... Image sparsify: 46% complete... Image sparsify: 48% complete... Image sparsify: 50% complete... Image sparsify: 51% complete... Image sparsify: 53% complete... Image sparsify: 54% complete... Image sparsify: 56% complete... Image sparsify: 57% complete... Image sparsify: 59% complete... Image sparsify: 60% complete... Image sparsify: 62% complete... Image sparsify: 64% complete... Image sparsify: 65% complete... Image sparsify: 67% complete... Image sparsify: 68% complete... Image sparsify: 70% complete... Image sparsify: 71% complete... Image sparsify: 73% complete... Image sparsify: 75% complete... Image sparsify: 76% complete... Image sparsify: 78% complete... Image sparsify: 79% complete... Image sparsify: 81% complete... Image sparsify: 82% complete... Image sparsify: 84% complete... Image sparsify: 85% complete... Image sparsify: 87% complete... Image sparsify: 89% complete... Image sparsify: 90% complete... Image sparsify: 92% complete... Image sparsify: 93% complete... Image sparsify: 95% complete... Image sparsify: 96% complete... Image sparsify: 98% complete... Image sparsify: 100% complete...done. 2023-12-24T18:00:15.440 INFO:tasks.workunit.client.0.smithi087.stderr:+ remove_images 2023-12-24T18:00:15.440 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:15.563 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:15.642 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:15.761 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:16.204 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:17.245 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:17.362 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:17.464 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:17.627 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:17.763 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:17.844 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:17.924 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:18.006 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:18.083 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:18.159 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:18.236 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:18.311 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:18.389 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:18.465 INFO:tasks.workunit.client.0.smithi087.stderr:+ rm -f /tmp/img1 /tmp/img1.new /tmp/img2 /tmp/img2.new /tmp/img3 /tmp/img3.new /tmp/img-diff1.new /tmp/img-diff2.new /tmp/img-diff3.new /tmp/img1.snap1 /tmp/img1.snap1 /tmp/img-diff1.snap1 2023-12-24T18:00:18.572 INFO:tasks.workunit.client.0.smithi087.stderr:+ test_locking 2023-12-24T18:00:18.573 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'testing locking...' 2023-12-24T18:00:18.573 INFO:tasks.workunit.client.0.smithi087.stdout:testing locking... 2023-12-24T18:00:18.574 INFO:tasks.workunit.client.0.smithi087.stderr:+ remove_images 2023-12-24T18:00:18.574 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:18.650 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:18.731 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:19.012 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:19.097 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:19.380 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:19.457 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:19.533 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:19.608 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:19.683 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:19.758 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:19.833 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:19.908 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:19.987 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:20.063 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:20.140 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:20.216 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:20.291 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:20.367 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create -s 1 test1 2023-12-24T18:00:20.393 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:00:20.396+0000 7f5c9dbfe540 -1 librbd: Forced V1 image creation. 2023-12-24T18:00:20.400 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd lock list test1 2023-12-24T18:00:20.401 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:00:20.401 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^0$' 2023-12-24T18:00:20.432 INFO:tasks.workunit.client.0.smithi087.stdout:0 2023-12-24T18:00:20.432 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd lock add test1 id 2023-12-24T18:00:20.470 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd lock list test1 2023-12-24T18:00:20.470 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep ' 1 ' 2023-12-24T18:00:20.501 INFO:tasks.workunit.client.0.smithi087.stdout:There is 1 exclusive lock on this image. 2023-12-24T18:00:20.502 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd lock list test1 2023-12-24T18:00:20.502 INFO:tasks.workunit.client.0.smithi087.stderr:++ tail -n 1 2023-12-24T18:00:20.502 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1;}' 2023-12-24T18:00:20.534 INFO:tasks.workunit.client.0.smithi087.stderr:+ LOCKER=client.7788 2023-12-24T18:00:20.534 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd lock remove test1 id client.7788 2023-12-24T18:00:21.108 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd lock list test1 2023-12-24T18:00:21.108 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:00:21.108 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^0$' 2023-12-24T18:00:21.143 INFO:tasks.workunit.client.0.smithi087.stdout:0 2023-12-24T18:00:21.143 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd lock add test1 id --shared tag 2023-12-24T18:00:21.186 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd lock list test1 2023-12-24T18:00:21.186 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep ' 1 ' 2023-12-24T18:00:21.218 INFO:tasks.workunit.client.0.smithi087.stdout:There is 1 shared lock on this image. 2023-12-24T18:00:21.218 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd lock add test1 id --shared tag 2023-12-24T18:00:21.258 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd lock list test1 2023-12-24T18:00:21.258 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep ' 2 ' 2023-12-24T18:00:21.290 INFO:tasks.workunit.client.0.smithi087.stdout:There are 2 shared locks on this image. 2023-12-24T18:00:21.290 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd lock add test1 id2 --shared tag 2023-12-24T18:00:21.330 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd lock list test1 2023-12-24T18:00:21.331 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep ' 3 ' 2023-12-24T18:00:21.362 INFO:tasks.workunit.client.0.smithi087.stdout:There are 3 shared locks on this image. 2023-12-24T18:00:21.363 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd lock list test1 2023-12-24T18:00:21.363 INFO:tasks.workunit.client.0.smithi087.stderr:+ tail -n 1 2023-12-24T18:00:21.363 INFO:tasks.workunit.client.0.smithi087.stderr:+ awk '{print $2, $1;}' 2023-12-24T18:00:21.363 INFO:tasks.workunit.client.0.smithi087.stderr:+ xargs rbd lock remove test1 2023-12-24T18:00:22.055 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info test1 2023-12-24T18:00:22.055 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -qE 'features:.*exclusive' 2023-12-24T18:00:22.088 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test1 2023-12-24T18:00:22.133 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T18:00:22.136 INFO:tasks.workunit.client.0.smithi087.stderr:+ test_thick_provision 2023-12-24T18:00:22.136 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'testing thick provision...' 2023-12-24T18:00:22.136 INFO:tasks.workunit.client.0.smithi087.stdout:testing thick provision... 2023-12-24T18:00:22.137 INFO:tasks.workunit.client.0.smithi087.stderr:+ remove_images 2023-12-24T18:00:22.137 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:22.211 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:22.286 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:22.363 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:22.441 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:22.517 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:22.593 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:22.871 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:22.948 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:23.023 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:23.099 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:23.173 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:23.249 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:23.327 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:23.404 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:23.481 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:23.560 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:23.636 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:23.713 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --thick-provision -s 64M test1 2023-12-24T18:00:23.741 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:00:23.744+0000 7f415f7ab540 -1 librbd: Forced V1 image creation. 2023-12-24T18:00:23.996 INFO:tasks.workunit.client.0.smithi087.stderr: Thick provisioning: 6% complete... Thick provisioning: 12% complete... Thick provisioning: 18% complete... Thick provisioning: 25% complete... Thick provisioning: 31% complete... Thick provisioning: 37% complete... Thick provisioning: 43% complete... Thick provisioning: 50% complete... Thick provisioning: 56% complete... Thick provisioning: 62% complete... Thick provisioning: 68% complete... Thick provisioning: 75% complete... Thick provisioning: 81% complete... Thick provisioning: 87% complete... Thick provisioning: 93% complete... Thick provisioning: 100% complete... Thick provisioning: 100% complete...done. 2023-12-24T18:00:24.007 INFO:tasks.workunit.client.0.smithi087.stderr:+ count=0 2023-12-24T18:00:24.008 INFO:tasks.workunit.client.0.smithi087.stderr:+ ret= 2023-12-24T18:00:24.008 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 0 -lt 10 ']' 2023-12-24T18:00:24.008 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd du 2023-12-24T18:00:24.008 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep test1 2023-12-24T18:00:24.008 INFO:tasks.workunit.client.0.smithi087.stderr:+ tr -s ' ' 2023-12-24T18:00:24.008 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^64 MiB' 2023-12-24T18:00:24.008 INFO:tasks.workunit.client.0.smithi087.stderr:+ cut -d ' ' -f 4-5 2023-12-24T18:00:24.037 INFO:tasks.workunit.client.0.smithi087.stderr:warning: fast-diff map is not enabled for test1. operation may be slow. 2023-12-24T18:00:24.041 INFO:tasks.workunit.client.0.smithi087.stdout:64 MiB 2023-12-24T18:00:24.042 INFO:tasks.workunit.client.0.smithi087.stderr:+ ret=0 2023-12-24T18:00:24.042 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 0 = 0 ']' 2023-12-24T18:00:24.042 INFO:tasks.workunit.client.0.smithi087.stderr:+ break 2023-12-24T18:00:24.042 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd du 2023-12-24T18:00:24.265 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:00:24.267+0000 7faf156a4700 0 --2- 172.21.15.87:0/2077058625 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55e3e86f7390 0x55e3e86ed2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-12-24T18:00:24.270 INFO:tasks.workunit.client.0.smithi087.stderr:warning: fast-diff map is not enabled for test1. operation may be slow. 2023-12-24T18:00:24.272 INFO:tasks.workunit.client.0.smithi087.stdout:NAME PROVISIONED USED 2023-12-24T18:00:24.272 INFO:tasks.workunit.client.0.smithi087.stdout:test1 64 MiB 64 MiB 2023-12-24T18:00:24.275 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 0 '!=' 0 ']' 2023-12-24T18:00:24.275 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test1 2023-12-24T18:00:24.345 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 6% complete... Removing image: 12% complete... Removing image: 18% complete... Removing image: 25% complete... Removing image: 31% complete... Removing image: 37% complete... Removing image: 43% complete... Removing image: 50% complete... Removing image: 56% complete... Removing image: 62% complete... Removing image: 68% complete... Removing image: 75% complete... Removing image: 81% complete... Removing image: 87% complete... Removing image: 93% complete... Removing image: 100% complete...done. 2023-12-24T18:00:24.350 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep test1 2023-12-24T18:00:24.350 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T18:00:24.352 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^0$' 2023-12-24T18:00:24.352 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:00:24.381 INFO:tasks.workunit.client.0.smithi087.stdout:0 2023-12-24T18:00:24.381 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --thick-provision -s 4G test1 2023-12-24T18:00:24.607 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:00:24.610+0000 7f3633ade700 0 --2- 172.21.15.87:0/2127134340 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x55d1f69f0d90 0x55d1f6a7ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-12-24T18:00:24.610 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:00:24.612+0000 7f363bfec540 -1 librbd: Forced V1 image creation. 2023-12-24T18:00:25.462 INFO:tasks.workunit.client.0.smithi087.stderr: Thick provisioning: 1% complete... Thick provisioning: 2% complete... Thick provisioning: 3% complete... Thick provisioning: 4% complete... Thick provisioning: 5% complete... Thick provisioning: 6% complete... Thick provisioning: 7% complete... Thick provisioning: 8% complete... Thick provisioning: 9% complete... Thick provisioning: 10% complete...2023-12-24T18:00:25.465+0000 7f36342df700 0 -- 172.21.15.87:0/2127134340 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x55d1f69f16a0 msgr2=0x55d1f6a7f770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-24T18:00:25.644 INFO:tasks.workunit.client.0.smithi087.stderr: Thick provisioning: 11% complete... Thick provisioning: 12% complete...2023-12-24T18:00:25.647+0000 7f3633ade700 0 -- 172.21.15.87:0/2127134340 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x55d1f6a93a50 msgr2=0x55d1f6a95e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-24T18:00:34.030 INFO:tasks.workunit.client.0.smithi087.stderr: Thick provisioning: 13% complete... Thick provisioning: 14% complete... Thick provisioning: 15% complete... Thick provisioning: 16% complete... Thick provisioning: 17% complete... Thick provisioning: 18% complete... Thick provisioning: 19% complete... Thick provisioning: 20% complete... Thick provisioning: 21% complete... Thick provisioning: 22% complete... Thick provisioning: 23% complete... Thick provisioning: 24% complete... Thick provisioning: 25% complete... Thick provisioning: 26% complete... Thick provisioning: 27% complete... Thick provisioning: 28% complete... Thick provisioning: 29% complete... Thick provisioning: 30% complete... Thick provisioning: 31% complete... Thick provisioning: 32% complete... Thick provisioning: 33% complete... Thick provisioning: 34% complete... Thick provisioning: 35% complete... Thick provisioning: 36% complete... Thick provisioning: 37% complete... Thick provisioning: 38% complete... Thick provisioning: 39% complete... Thick provisioning: 40% complete... Thick provisioning: 41% complete... Thick provisioning: 42% complete... Thick provisioning: 43% complete... Thick provisioning: 44% complete... Thick provisioning: 45% complete... Thick provisioning: 46% complete... Thick provisioning: 47% complete... Thick provisioning: 48% complete... Thick provisioning: 49% complete... Thick provisioning: 50% complete... Thick provisioning: 51% complete... Thick provisioning: 52% complete... Thick provisioning: 53% complete... Thick provisioning: 54% complete... Thick provisioning: 55% complete... Thick provisioning: 56% complete... Thick provisioning: 57% complete... Thick provisioning: 58% complete... Thick provisioning: 59% complete... Thick provisioning: 60% complete... Thick provisioning: 61% complete... Thick provisioning: 62% complete... Thick provisioning: 63% complete... Thick provisioning: 64% complete... Thick provisioning: 65% complete... Thick provisioning: 66% complete... Thick provisioning: 67% complete... Thick provisioning: 68% complete... Thick provisioning: 69% complete... Thick provisioning: 70% complete... Thick provisioning: 71% complete... Thick provisioning: 72% complete... Thick provisioning: 73% complete... Thick provisioning: 74% complete... Thick provisioning: 75% complete... Thick provisioning: 76% complete... Thick provisioning: 77% complete... Thick provisioning: 78% complete... Thick provisioning: 79% complete... Thick provisioning: 80% complete... Thick provisioning: 81% complete... Thick provisioning: 82% complete... Thick provisioning: 83% complete... Thick provisioning: 84% complete... Thick provisioning: 85% complete... Thick provisioning: 86% complete... Thick provisioning: 87% complete... Thick provisioning: 88% complete... Thick provisioning: 89% complete... Thick provisioning: 90% complete... Thick provisioning: 91% complete... Thick provisioning: 92% complete... Thick provisioning: 93% complete... Thick provisioning: 94% complete... Thick provisioning: 95% complete... Thick provisioning: 96% complete... Thick provisioning: 97% complete... Thick provisioning: 98% complete... Thick provisioning: 99% complete... Thick provisioning: 100% complete... Thick provisioning: 100% complete...done. 2023-12-24T18:00:34.036 INFO:tasks.workunit.client.0.smithi087.stderr:+ count=0 2023-12-24T18:00:34.037 INFO:tasks.workunit.client.0.smithi087.stderr:+ ret= 2023-12-24T18:00:34.037 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 0 -lt 10 ']' 2023-12-24T18:00:34.037 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd du 2023-12-24T18:00:34.037 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep test1 2023-12-24T18:00:34.037 INFO:tasks.workunit.client.0.smithi087.stderr:+ tr -s ' ' 2023-12-24T18:00:34.037 INFO:tasks.workunit.client.0.smithi087.stderr:+ cut -d ' ' -f 4-5 2023-12-24T18:00:34.037 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^4 GiB' 2023-12-24T18:00:34.064 INFO:tasks.workunit.client.0.smithi087.stderr:warning: fast-diff map is not enabled for test1. operation may be slow. 2023-12-24T18:00:34.110 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:00:34.113+0000 7f0bc4430700 0 -- 172.21.15.87:0/1070476649 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x555f17aba870 msgr2=0x555f17abcc50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:00:34.121 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:00:34.123+0000 7f0bc4430700 0 -- 172.21.15.87:0/1070476649 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f0ba803a250 msgr2=0x7f0ba803c670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:00:34.235 INFO:tasks.workunit.client.0.smithi087.stdout:4 GiB 2023-12-24T18:00:34.236 INFO:tasks.workunit.client.0.smithi087.stderr:+ ret=0 2023-12-24T18:00:34.236 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 0 = 0 ']' 2023-12-24T18:00:34.236 INFO:tasks.workunit.client.0.smithi087.stderr:+ break 2023-12-24T18:00:34.236 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd du 2023-12-24T18:00:34.264 INFO:tasks.workunit.client.0.smithi087.stderr:warning: fast-diff map is not enabled for test1. operation may be slow. 2023-12-24T18:00:34.276 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:00:34.278+0000 7f20e61bf700 0 -- 172.21.15.87:0/3370971709 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x5617f4716ba0 msgr2=0x5617f4726550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:00:34.278 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:00:34.281+0000 7f20e61bf700 0 -- 172.21.15.87:0/3370971709 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7f20c803a250 msgr2=0x7f20c803c670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:00:34.391 INFO:tasks.workunit.client.0.smithi087.stdout:NAME PROVISIONED USED 2023-12-24T18:00:34.391 INFO:tasks.workunit.client.0.smithi087.stdout:test1 4 GiB 4 GiB 2023-12-24T18:00:34.393 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 0 '!=' 0 ']' 2023-12-24T18:00:34.393 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test1 2023-12-24T18:00:34.506 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete...2023-12-24T18:00:34.509+0000 7fb0bf848700 0 -- 172.21.15.87:0/1217796999 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x55f42a1f7720 msgr2=0x55f42a1f9b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:00:34.524 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 11% complete... Removing image: 12% complete...2023-12-24T18:00:34.527+0000 7fb0c0049700 0 -- 172.21.15.87:0/1217796999 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x55f42a14fb50 msgr2=0x55f42a1f6cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:00:35.060 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:00:35.062+0000 7f6d99bae700 -1 reset not still connected to 0x562cfe5e3e10 2023-12-24T18:00:35.311 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-24T18:00:35.314 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T18:00:35.314 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep test1 2023-12-24T18:00:35.314 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:00:35.314 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^0$' 2023-12-24T18:00:35.340 INFO:tasks.workunit.client.0.smithi087.stdout:0 2023-12-24T18:00:35.340 INFO:tasks.workunit.client.0.smithi087.stderr:+ RBD_CREATE_ARGS='--image-format 2' 2023-12-24T18:00:35.340 INFO:tasks.workunit.client.0.smithi087.stderr:+ test_others 2023-12-24T18:00:35.341 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'testing import, export, resize, and snapshots...' 2023-12-24T18:00:35.341 INFO:tasks.workunit.client.0.smithi087.stdout:testing import, export, resize, and snapshots... 2023-12-24T18:00:35.341 INFO:tasks.workunit.client.0.smithi087.stderr:+ TMP_FILES='/tmp/img1 /tmp/img1.new /tmp/img2 /tmp/img2.new /tmp/img3 /tmp/img3.new /tmp/img-diff1.new /tmp/img-diff2.new /tmp/img-diff3.new /tmp/img1.snap1 /tmp/img1.snap1 /tmp/img-diff1.snap1' 2023-12-24T18:00:35.341 INFO:tasks.workunit.client.0.smithi087.stderr:+ remove_images 2023-12-24T18:00:35.341 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:35.417 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:35.493 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:35.572 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:35.650 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:35.729 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:35.802 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:35.879 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:35.955 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:36.031 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:36.106 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:36.382 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:36.457 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:36.535 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:36.610 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:36.686 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:36.763 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:36.841 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:36.920 INFO:tasks.workunit.client.0.smithi087.stderr:+ rm -f /tmp/img1 /tmp/img1.new /tmp/img2 /tmp/img2.new /tmp/img3 /tmp/img3.new /tmp/img-diff1.new /tmp/img-diff2.new /tmp/img-diff3.new /tmp/img1.snap1 /tmp/img1.snap1 /tmp/img-diff1.snap1 2023-12-24T18:00:36.920 INFO:tasks.workunit.client.0.smithi087.stderr:+ dd if=/bin/sh of=/tmp/img1 bs=1k count=1 seek=10 2023-12-24T18:00:36.921 INFO:tasks.workunit.client.0.smithi087.stderr:1+0 records in 2023-12-24T18:00:36.922 INFO:tasks.workunit.client.0.smithi087.stderr:1+0 records out 2023-12-24T18:00:36.922 INFO:tasks.workunit.client.0.smithi087.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 3.8247e-05 s, 26.8 MB/s 2023-12-24T18:00:36.922 INFO:tasks.workunit.client.0.smithi087.stderr:+ dd if=/bin/dd of=/tmp/img1 bs=1k count=10 seek=100 2023-12-24T18:00:36.922 INFO:tasks.workunit.client.0.smithi087.stderr:10+0 records in 2023-12-24T18:00:36.922 INFO:tasks.workunit.client.0.smithi087.stderr:10+0 records out 2023-12-24T18:00:36.922 INFO:tasks.workunit.client.0.smithi087.stderr:10240 bytes (10 kB, 10 KiB) copied, 5.1033e-05 s, 201 MB/s 2023-12-24T18:00:36.922 INFO:tasks.workunit.client.0.smithi087.stderr:+ dd if=/bin/rm of=/tmp/img1 bs=1k count=100 seek=1000 2023-12-24T18:00:36.923 INFO:tasks.workunit.client.0.smithi087.stderr:70+1 records in 2023-12-24T18:00:36.923 INFO:tasks.workunit.client.0.smithi087.stderr:70+1 records out 2023-12-24T18:00:36.923 INFO:tasks.workunit.client.0.smithi087.stderr:72208 bytes (72 kB, 71 KiB) copied, 0.000193401 s, 373 MB/s 2023-12-24T18:00:36.923 INFO:tasks.workunit.client.0.smithi087.stderr:+ dd if=/bin/ls of=/tmp/img1 bs=1k seek=10000 2023-12-24T18:00:36.924 INFO:tasks.workunit.client.0.smithi087.stderr:140+1 records in 2023-12-24T18:00:36.924 INFO:tasks.workunit.client.0.smithi087.stderr:140+1 records out 2023-12-24T18:00:36.924 INFO:tasks.workunit.client.0.smithi087.stderr:143376 bytes (143 kB, 140 KiB) copied, 0.000332565 s, 431 MB/s 2023-12-24T18:00:36.924 INFO:tasks.workunit.client.0.smithi087.stderr:+ dd if=/bin/ln of=/tmp/img1 bs=1k seek=100000 2023-12-24T18:00:36.925 INFO:tasks.workunit.client.0.smithi087.stderr:70+1 records in 2023-12-24T18:00:36.925 INFO:tasks.workunit.client.0.smithi087.stderr:70+1 records out 2023-12-24T18:00:36.925 INFO:tasks.workunit.client.0.smithi087.stderr:72312 bytes (72 kB, 71 KiB) copied, 0.000170414 s, 424 MB/s 2023-12-24T18:00:36.925 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd import --image-format 2 /tmp/img1 testimg1 2023-12-24T18:00:37.053 INFO:tasks.workunit.client.0.smithi087.stderr: Importing image: 4% complete... Importing image: 8% complete... Importing image: 12% complete... Importing image: 16% complete... Importing image: 20% complete... Importing image: 24% complete... Importing image: 28% complete... Importing image: 32% complete... Importing image: 36% complete... Importing image: 40% complete... Importing image: 45% complete... Importing image: 49% complete... Importing image: 53% complete... Importing image: 57% complete... Importing image: 61% complete... Importing image: 65% complete... Importing image: 69% complete... Importing image: 73% complete... Importing image: 77% complete... Importing image: 81% complete... Importing image: 85% complete... Importing image: 90% complete... Importing image: 94% complete... Importing image: 98% complete... Importing image: 100% complete...done. 2023-12-24T18:00:37.055 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd resize testimg1 --size=256 --allow-shrink 2023-12-24T18:00:37.099 INFO:tasks.workunit.client.0.smithi087.stderr: Resizing image: 100% complete...done. 2023-12-24T18:00:37.106 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd export testimg1 /tmp/img2 2023-12-24T18:00:37.206 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 1% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 73% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 87% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 98% complete... Exporting image: 100% complete...done. 2023-12-24T18:00:37.207 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create testimg1 --snap=snap1 2023-12-24T18:00:38.235 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T18:00:38.244 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd resize testimg1 --size=128 2023-12-24T18:00:38.281 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: shrinking an image is only allowed with the --allow-shrink flag 2023-12-24T18:00:38.285 INFO:tasks.workunit.client.0.smithi087.stderr:+ true 2023-12-24T18:00:38.286 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd resize testimg1 --size=128 --allow-shrink 2023-12-24T18:00:38.330 INFO:tasks.workunit.client.0.smithi087.stderr: Resizing image: 50% complete... Resizing image: 51% complete... Resizing image: 53% complete... Resizing image: 54% complete... Resizing image: 56% complete... Resizing image: 57% complete... Resizing image: 59% complete... Resizing image: 60% complete... Resizing image: 62% complete... Resizing image: 64% complete... Resizing image: 65% complete... Resizing image: 67% complete... Resizing image: 68% complete... Resizing image: 70% complete... Resizing image: 71% complete... Resizing image: 73% complete... Resizing image: 75% complete... Resizing image: 76% complete... Resizing image: 78% complete... Resizing image: 79% complete... Resizing image: 81% complete... Resizing image: 82% complete... Resizing image: 84% complete... Resizing image: 85% complete... Resizing image: 87% complete... Resizing image: 89% complete... Resizing image: 90% complete... Resizing image: 92% complete... Resizing image: 93% complete... Resizing image: 95% complete... Resizing image: 96% complete... Resizing image: 98% complete... Resizing image: 100% complete...done. 2023-12-24T18:00:38.338 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd export testimg1 /tmp/img3 2023-12-24T18:00:38.614 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-24T18:00:38.617 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info testimg1 2023-12-24T18:00:38.617 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'size 128 MiB' 2023-12-24T18:00:38.650 INFO:tasks.workunit.client.0.smithi087.stdout: size 128 MiB in 32 objects 2023-12-24T18:00:38.651 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info --snap=snap1 testimg1 2023-12-24T18:00:38.651 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'size 256 MiB' 2023-12-24T18:00:38.686 INFO:tasks.workunit.client.0.smithi087.stdout: size 256 MiB in 64 objects 2023-12-24T18:00:38.686 INFO:tasks.workunit.client.0.smithi087.stderr:+ rm -rf /tmp/diff-testimg1-1 /tmp/diff-testimg1-2 2023-12-24T18:00:38.687 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd export-diff testimg1 --snap=snap1 /tmp/diff-testimg1-1 2023-12-24T18:00:38.752 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 3% complete... Exporting image: 37% complete... Exporting image: 100% complete...done. 2023-12-24T18:00:38.755 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd export-diff testimg1 --from-snap=snap1 /tmp/diff-testimg1-2 2023-12-24T18:00:38.800 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 100% complete...done. 2023-12-24T18:00:38.802 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 --size=1 testimg-diff1 2023-12-24T18:00:38.846 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd import-diff --sparse-size 8K /tmp/diff-testimg1-1 testimg-diff1 2023-12-24T18:00:39.242 INFO:tasks.workunit.client.0.smithi087.stderr: Importing image diff: 22% complete... Importing image diff: 63% complete... Importing image diff: 99% complete... Importing image diff: 100% complete...done. 2023-12-24T18:00:39.254 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd import-diff --sparse-size 8K /tmp/diff-testimg1-2 testimg-diff1 2023-12-24T18:00:39.298 INFO:tasks.workunit.client.0.smithi087.stderr: Importing image diff: 68% complete... Importing image diff: 96% complete... Importing image diff: 100% complete...done. 2023-12-24T18:00:39.306 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info testimg1 2023-12-24T18:00:39.306 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'size 128 MiB' 2023-12-24T18:00:39.340 INFO:tasks.workunit.client.0.smithi087.stdout: size 128 MiB in 32 objects 2023-12-24T18:00:39.341 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info --snap=snap1 testimg1 2023-12-24T18:00:39.341 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'size 256 MiB' 2023-12-24T18:00:39.376 INFO:tasks.workunit.client.0.smithi087.stdout: size 256 MiB in 64 objects 2023-12-24T18:00:39.376 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info testimg-diff1 2023-12-24T18:00:39.376 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'size 128 MiB' 2023-12-24T18:00:39.410 INFO:tasks.workunit.client.0.smithi087.stdout: size 128 MiB in 32 objects 2023-12-24T18:00:39.411 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info --snap=snap1 testimg-diff1 2023-12-24T18:00:39.411 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'size 256 MiB' 2023-12-24T18:00:39.446 INFO:tasks.workunit.client.0.smithi087.stdout: size 256 MiB in 64 objects 2023-12-24T18:00:39.446 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd copy testimg1 --snap=snap1 testimg2 2023-12-24T18:00:39.530 INFO:tasks.workunit.client.0.smithi087.stderr: Image copy: 3% complete... Image copy: 37% complete... Image copy: 100% complete... Image copy: 100% complete...done. 2023-12-24T18:00:39.533 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd copy testimg1 testimg3 2023-12-24T18:00:39.633 INFO:tasks.workunit.client.0.smithi087.stderr: Image copy: 3% complete... Image copy: 6% complete... Image copy: 9% complete... Image copy: 12% complete... Image copy: 15% complete... Image copy: 18% complete... Image copy: 21% complete... Image copy: 25% complete... Image copy: 28% complete... Image copy: 31% complete... Image copy: 34% complete... Image copy: 37% complete... Image copy: 40% complete... Image copy: 43% complete... Image copy: 46% complete... Image copy: 50% complete... Image copy: 53% complete... Image copy: 56% complete... Image copy: 59% complete... Image copy: 62% complete... Image copy: 65% complete... Image copy: 68% complete... Image copy: 71% complete... Image copy: 75% complete... Image copy: 78% complete... Image copy: 81% complete... Image copy: 84% complete... Image copy: 87% complete... Image copy: 90% complete... Image copy: 93% complete... Image copy: 96% complete... Image copy: 100% complete... Image copy: 100% complete...done. 2023-12-24T18:00:39.636 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd copy testimg-diff1 --sparse-size 768K --snap=snap1 testimg-diff2 2023-12-24T18:00:39.739 INFO:tasks.workunit.client.0.smithi087.stderr: Image copy: 3% complete... Image copy: 37% complete... Image copy: 100% complete... Image copy: 100% complete...done. 2023-12-24T18:00:39.742 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd copy testimg-diff1 --sparse-size 768K testimg-diff3 2023-12-24T18:00:39.850 INFO:tasks.workunit.client.0.smithi087.stderr: Image copy: 3% complete... Image copy: 6% complete... Image copy: 9% complete... Image copy: 12% complete... Image copy: 15% complete... Image copy: 18% complete... Image copy: 21% complete... Image copy: 25% complete... Image copy: 28% complete... Image copy: 31% complete... Image copy: 34% complete... Image copy: 37% complete... Image copy: 40% complete... Image copy: 43% complete... Image copy: 46% complete... Image copy: 50% complete... Image copy: 53% complete... Image copy: 56% complete... Image copy: 59% complete... Image copy: 62% complete... Image copy: 65% complete... Image copy: 68% complete... Image copy: 71% complete... Image copy: 75% complete... Image copy: 78% complete... Image copy: 81% complete... Image copy: 84% complete... Image copy: 87% complete... Image copy: 90% complete... Image copy: 93% complete... Image copy: 96% complete... Image copy: 100% complete... Image copy: 100% complete...done. 2023-12-24T18:00:39.853 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info testimg2 2023-12-24T18:00:39.853 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'size 256 MiB' 2023-12-24T18:00:39.888 INFO:tasks.workunit.client.0.smithi087.stdout: size 256 MiB in 64 objects 2023-12-24T18:00:39.888 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info testimg3 2023-12-24T18:00:39.888 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'size 128 MiB' 2023-12-24T18:00:39.920 INFO:tasks.workunit.client.0.smithi087.stdout: size 128 MiB in 32 objects 2023-12-24T18:00:39.921 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info testimg-diff2 2023-12-24T18:00:39.921 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'size 256 MiB' 2023-12-24T18:00:39.955 INFO:tasks.workunit.client.0.smithi087.stdout: size 256 MiB in 64 objects 2023-12-24T18:00:39.955 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info testimg-diff3 2023-12-24T18:00:39.955 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'size 128 MiB' 2023-12-24T18:00:39.988 INFO:tasks.workunit.client.0.smithi087.stdout: size 128 MiB in 32 objects 2023-12-24T18:00:39.988 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd deep copy testimg1 testimg4 2023-12-24T18:00:40.303 INFO:tasks.workunit.client.0.smithi087.stderr: Image deep copy: 1% complete... Image deep copy: 3% complete... Image deep copy: 4% complete... Image deep copy: 6% complete... Image deep copy: 7% complete... Image deep copy: 9% complete... Image deep copy: 10% complete... Image deep copy: 12% complete... Image deep copy: 14% complete... Image deep copy: 15% complete... Image deep copy: 17% complete... Image deep copy: 18% complete... Image deep copy: 20% complete... Image deep copy: 21% complete... Image deep copy: 23% complete... Image deep copy: 25% complete... Image deep copy: 26% complete... Image deep copy: 28% complete... Image deep copy: 29% complete... Image deep copy: 31% complete... Image deep copy: 32% complete... Image deep copy: 34% complete... Image deep copy: 35% complete... Image deep copy: 37% complete... Image deep copy: 39% complete... Image deep copy: 40% complete... Image deep copy: 42% complete... Image deep copy: 43% complete... Image deep copy: 45% complete... Image deep copy: 46% complete... Image deep copy: 48% complete... Image deep copy: 50% complete... Image deep copy: 51% complete... Image deep copy: 53% complete... Image deep copy: 54% complete... Image deep copy: 56% complete... Image deep copy: 57% complete... Image deep copy: 59% complete... Image deep copy: 60% complete... Image deep copy: 62% complete... Image deep copy: 64% complete... Image deep copy: 65% complete... Image deep copy: 67% complete... Image deep copy: 68% complete... Image deep copy: 70% complete... Image deep copy: 71% complete... Image deep copy: 73% complete... Image deep copy: 75% complete... Image deep copy: 76% complete... Image deep copy: 78% complete... Image deep copy: 79% complete... Image deep copy: 81% complete... Image deep copy: 82% complete... Image deep copy: 84% complete... Image deep copy: 85% complete... Image deep copy: 87% complete... Image deep copy: 89% complete... Image deep copy: 90% complete... Image deep copy: 92% complete... Image deep copy: 93% complete... Image deep copy: 95% complete... Image deep copy: 96% complete... Image deep copy: 98% complete... Image deep copy: 100% complete... Image deep copy: 100% complete...done. 2023-12-24T18:00:40.306 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd deep copy testimg1 --snap=snap1 testimg5 2023-12-24T18:00:41.297 INFO:tasks.workunit.client.0.smithi087.stderr: Image deep copy: 1% complete... Image deep copy: 3% complete... Image deep copy: 4% complete... Image deep copy: 6% complete... Image deep copy: 7% complete... Image deep copy: 9% complete... Image deep copy: 10% complete... Image deep copy: 12% complete... Image deep copy: 14% complete... Image deep copy: 15% complete... Image deep copy: 17% complete... Image deep copy: 18% complete... Image deep copy: 20% complete... Image deep copy: 21% complete... Image deep copy: 23% complete... Image deep copy: 25% complete... Image deep copy: 26% complete... Image deep copy: 28% complete... Image deep copy: 29% complete... Image deep copy: 31% complete... Image deep copy: 32% complete... Image deep copy: 34% complete... Image deep copy: 35% complete... Image deep copy: 37% complete... Image deep copy: 39% complete... Image deep copy: 40% complete... Image deep copy: 42% complete... Image deep copy: 43% complete... Image deep copy: 45% complete... Image deep copy: 46% complete... Image deep copy: 48% complete... Image deep copy: 50% complete... Image deep copy: 51% complete... Image deep copy: 53% complete... Image deep copy: 54% complete... Image deep copy: 56% complete... Image deep copy: 57% complete... Image deep copy: 59% complete... Image deep copy: 60% complete... Image deep copy: 62% complete... Image deep copy: 64% complete... Image deep copy: 65% complete... Image deep copy: 67% complete... Image deep copy: 68% complete... Image deep copy: 70% complete... Image deep copy: 71% complete... Image deep copy: 73% complete... Image deep copy: 75% complete... Image deep copy: 76% complete... Image deep copy: 78% complete... Image deep copy: 79% complete... Image deep copy: 81% complete... Image deep copy: 82% complete... Image deep copy: 84% complete... Image deep copy: 85% complete... Image deep copy: 87% complete... Image deep copy: 89% complete... Image deep copy: 90% complete... Image deep copy: 92% complete... Image deep copy: 93% complete... Image deep copy: 95% complete... Image deep copy: 96% complete... Image deep copy: 98% complete... Image deep copy: 100% complete... Image deep copy: 100% complete...done. 2023-12-24T18:00:41.300 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info testimg4 2023-12-24T18:00:41.300 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'size 128 MiB' 2023-12-24T18:00:41.336 INFO:tasks.workunit.client.0.smithi087.stdout: size 128 MiB in 32 objects 2023-12-24T18:00:41.336 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info testimg5 2023-12-24T18:00:41.336 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'size 256 MiB' 2023-12-24T18:00:41.370 INFO:tasks.workunit.client.0.smithi087.stdout: size 256 MiB in 64 objects 2023-12-24T18:00:41.371 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap ls testimg4 2023-12-24T18:00:41.371 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -v SNAPID 2023-12-24T18:00:41.371 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:00:41.371 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 1 2023-12-24T18:00:41.405 INFO:tasks.workunit.client.0.smithi087.stdout:1 2023-12-24T18:00:41.406 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap ls testimg4 2023-12-24T18:00:41.406 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '.*snap1.*' 2023-12-24T18:00:41.440 INFO:tasks.workunit.client.0.smithi087.stdout: 26 snap1 256 MiB Sun Dec 24 18:00:40 2023 2023-12-24T18:00:41.441 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd export testimg1 /tmp/img1.new 2023-12-24T18:00:41.514 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-24T18:00:41.517 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd export testimg2 /tmp/img2.new 2023-12-24T18:00:41.618 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 1% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 73% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 87% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 98% complete... Exporting image: 100% complete...done. 2023-12-24T18:00:41.621 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd export testimg3 /tmp/img3.new 2023-12-24T18:00:41.691 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-24T18:00:41.694 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd export testimg-diff1 /tmp/img-diff1.new 2023-12-24T18:00:41.766 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-24T18:00:41.769 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd export testimg-diff2 /tmp/img-diff2.new 2023-12-24T18:00:41.864 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 1% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 73% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 87% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 98% complete... Exporting image: 100% complete...done. 2023-12-24T18:00:41.868 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd export testimg-diff3 /tmp/img-diff3.new 2023-12-24T18:00:41.939 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-24T18:00:41.942 INFO:tasks.workunit.client.0.smithi087.stderr:+ cmp /tmp/img2 /tmp/img2.new 2023-12-24T18:00:42.136 INFO:tasks.workunit.client.0.smithi087.stderr:+ cmp /tmp/img3 /tmp/img3.new 2023-12-24T18:00:42.231 INFO:tasks.workunit.client.0.smithi087.stderr:+ cmp /tmp/img2 /tmp/img-diff2.new 2023-12-24T18:00:42.388 INFO:tasks.workunit.client.0.smithi087.stderr:+ cmp /tmp/img3 /tmp/img-diff3.new 2023-12-24T18:00:42.465 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap rollback --snap=snap1 testimg1 2023-12-24T18:00:42.552 INFO:tasks.workunit.client.0.smithi087.stderr: Rolling back to snapshot: 1% complete... Rolling back to snapshot: 3% complete... Rolling back to snapshot: 4% complete... Rolling back to snapshot: 6% complete... Rolling back to snapshot: 7% complete... Rolling back to snapshot: 9% complete... Rolling back to snapshot: 10% complete... Rolling back to snapshot: 12% complete... Rolling back to snapshot: 14% complete... Rolling back to snapshot: 15% complete... Rolling back to snapshot: 17% complete... Rolling back to snapshot: 18% complete... Rolling back to snapshot: 20% complete... Rolling back to snapshot: 21% complete... Rolling back to snapshot: 23% complete... Rolling back to snapshot: 25% complete... Rolling back to snapshot: 26% complete... Rolling back to snapshot: 28% complete... Rolling back to snapshot: 29% complete... Rolling back to snapshot: 31% complete... Rolling back to snapshot: 32% complete... Rolling back to snapshot: 34% complete... Rolling back to snapshot: 35% complete... Rolling back to snapshot: 37% complete... Rolling back to snapshot: 39% complete... Rolling back to snapshot: 40% complete... Rolling back to snapshot: 42% complete... Rolling back to snapshot: 43% complete... Rolling back to snapshot: 45% complete... Rolling back to snapshot: 46% complete... Rolling back to snapshot: 48% complete... Rolling back to snapshot: 50% complete... Rolling back to snapshot: 51% complete... Rolling back to snapshot: 53% complete... Rolling back to snapshot: 54% complete... Rolling back to snapshot: 56% complete... Rolling back to snapshot: 57% complete... Rolling back to snapshot: 59% complete... Rolling back to snapshot: 60% complete... Rolling back to snapshot: 62% complete... Rolling back to snapshot: 64% complete... Rolling back to snapshot: 65% complete... Rolling back to snapshot: 67% complete... Rolling back to snapshot: 68% complete... Rolling back to snapshot: 70% complete... Rolling back to snapshot: 71% complete... Rolling back to snapshot: 73% complete... Rolling back to snapshot: 75% complete... Rolling back to snapshot: 76% complete... Rolling back to snapshot: 78% complete... Rolling back to snapshot: 79% complete... Rolling back to snapshot: 81% complete... Rolling back to snapshot: 82% complete... Rolling back to snapshot: 84% complete... Rolling back to snapshot: 85% complete... Rolling back to snapshot: 87% complete... Rolling back to snapshot: 89% complete... Rolling back to snapshot: 90% complete... Rolling back to snapshot: 92% complete... Rolling back to snapshot: 93% complete... Rolling back to snapshot: 95% complete... Rolling back to snapshot: 96% complete... Rolling back to snapshot: 98% complete... Rolling back to snapshot: 100% complete...done. 2023-12-24T18:00:42.561 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap rollback --snap=snap1 testimg-diff1 2023-12-24T18:00:42.649 INFO:tasks.workunit.client.0.smithi087.stderr: Rolling back to snapshot: 1% complete... Rolling back to snapshot: 3% complete... Rolling back to snapshot: 4% complete... Rolling back to snapshot: 6% complete... Rolling back to snapshot: 7% complete... Rolling back to snapshot: 9% complete... Rolling back to snapshot: 10% complete... Rolling back to snapshot: 12% complete... Rolling back to snapshot: 14% complete... Rolling back to snapshot: 15% complete... Rolling back to snapshot: 17% complete... Rolling back to snapshot: 18% complete... Rolling back to snapshot: 20% complete... Rolling back to snapshot: 21% complete... Rolling back to snapshot: 23% complete... Rolling back to snapshot: 25% complete... Rolling back to snapshot: 26% complete... Rolling back to snapshot: 28% complete... Rolling back to snapshot: 29% complete... Rolling back to snapshot: 31% complete... Rolling back to snapshot: 32% complete... Rolling back to snapshot: 34% complete... Rolling back to snapshot: 35% complete... Rolling back to snapshot: 37% complete... Rolling back to snapshot: 39% complete... Rolling back to snapshot: 40% complete... Rolling back to snapshot: 42% complete... Rolling back to snapshot: 43% complete... Rolling back to snapshot: 45% complete... Rolling back to snapshot: 46% complete... Rolling back to snapshot: 48% complete... Rolling back to snapshot: 50% complete... Rolling back to snapshot: 51% complete... Rolling back to snapshot: 53% complete... Rolling back to snapshot: 54% complete... Rolling back to snapshot: 56% complete... Rolling back to snapshot: 57% complete... Rolling back to snapshot: 59% complete... Rolling back to snapshot: 60% complete... Rolling back to snapshot: 62% complete... Rolling back to snapshot: 64% complete... Rolling back to snapshot: 65% complete... Rolling back to snapshot: 67% complete... Rolling back to snapshot: 68% complete... Rolling back to snapshot: 70% complete... Rolling back to snapshot: 71% complete... Rolling back to snapshot: 73% complete... Rolling back to snapshot: 75% complete... Rolling back to snapshot: 76% complete... Rolling back to snapshot: 78% complete... Rolling back to snapshot: 79% complete... Rolling back to snapshot: 81% complete... Rolling back to snapshot: 82% complete... Rolling back to snapshot: 84% complete... Rolling back to snapshot: 85% complete... Rolling back to snapshot: 87% complete... Rolling back to snapshot: 89% complete... Rolling back to snapshot: 90% complete... Rolling back to snapshot: 92% complete... Rolling back to snapshot: 93% complete... Rolling back to snapshot: 95% complete... Rolling back to snapshot: 96% complete... Rolling back to snapshot: 98% complete... Rolling back to snapshot: 100% complete...done. 2023-12-24T18:00:42.657 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info testimg1 2023-12-24T18:00:42.658 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'size 256 MiB' 2023-12-24T18:00:42.691 INFO:tasks.workunit.client.0.smithi087.stdout: size 256 MiB in 64 objects 2023-12-24T18:00:42.691 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info testimg-diff1 2023-12-24T18:00:42.691 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'size 256 MiB' 2023-12-24T18:00:42.725 INFO:tasks.workunit.client.0.smithi087.stdout: size 256 MiB in 64 objects 2023-12-24T18:00:42.725 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd export testimg1 /tmp/img1.snap1 2023-12-24T18:00:42.815 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 1% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 73% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 87% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 98% complete... Exporting image: 100% complete...done. 2023-12-24T18:00:42.819 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd export testimg-diff1 /tmp/img-diff1.snap1 2023-12-24T18:00:42.924 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 1% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 73% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 87% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 98% complete... Exporting image: 100% complete...done. 2023-12-24T18:00:42.927 INFO:tasks.workunit.client.0.smithi087.stderr:+ cmp /tmp/img2 /tmp/img1.snap1 2023-12-24T18:00:43.075 INFO:tasks.workunit.client.0.smithi087.stderr:+ cmp /tmp/img2 /tmp/img-diff1.snap1 2023-12-24T18:00:43.221 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm testimg2 2023-12-24T18:00:43.330 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 1% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 12% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 23% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 37% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 48% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 62% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 73% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 87% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 98% complete...2023-12-24T18:00:43.333+0000 7fa1c2cfb700 0 -- 172.21.15.87:0/2081263640 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fa1a0008740 msgr2=0x7fa1a000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-24T18:00:43.341 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T18:00:43.344 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm testimg3 2023-12-24T18:00:43.438 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete...2023-12-24T18:00:43.441+0000 7f023d25b700 0 -- 172.21.15.87:0/2761595491 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x55e22b4d3da0 msgr2=0x55e22b575d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:00:43.447 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T18:00:43.450 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create testimg2 -s 0 2023-12-24T18:00:43.477 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:00:43.480+0000 7f9acb45a540 -1 librbd: Forced V1 image creation. 2023-12-24T18:00:43.483 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd cp testimg2 testimg3 2023-12-24T18:00:43.535 INFO:tasks.workunit.client.0.smithi087.stderr: Image copy: 100% complete...done. 2023-12-24T18:00:43.537 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd deep cp testimg2 testimg6 2023-12-24T18:00:43.592 INFO:tasks.workunit.client.0.smithi087.stderr: Image deep copy: 100% complete...done. 2023-12-24T18:00:43.594 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap rm --snap=snap1 testimg1 2023-12-24T18:00:44.248 INFO:tasks.workunit.client.0.smithi087.stderr: Removing snap: 100% complete...done. 2023-12-24T18:00:44.265 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap rm --snap=snap1 testimg-diff1 2023-12-24T18:00:45.254 INFO:tasks.workunit.client.0.smithi087.stderr: Removing snap: 100% complete...done. 2023-12-24T18:00:45.266 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info --snap=snap1 testimg1 2023-12-24T18:00:45.266 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'error setting snapshot context: (2) No such file or directory' 2023-12-24T18:00:45.303 INFO:tasks.workunit.client.0.smithi087.stdout:error setting snapshot context: (2) No such file or directory 2023-12-24T18:00:45.304 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info --snap=snap1 testimg-diff1 2023-12-24T18:00:45.304 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'error setting snapshot context: (2) No such file or directory' 2023-12-24T18:00:45.339 INFO:tasks.workunit.client.0.smithi087.stdout:error setting snapshot context: (2) No such file or directory 2023-12-24T18:00:45.339 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd sparsify testimg1 2023-12-24T18:00:45.404 INFO:tasks.workunit.client.0.smithi087.stderr: Image sparsify: 1% complete... Image sparsify: 3% complete... Image sparsify: 4% complete... Image sparsify: 6% complete... Image sparsify: 7% complete... Image sparsify: 9% complete... Image sparsify: 10% complete... Image sparsify: 12% complete... Image sparsify: 14% complete... Image sparsify: 15% complete... Image sparsify: 17% complete... Image sparsify: 18% complete... Image sparsify: 20% complete... Image sparsify: 21% complete... Image sparsify: 23% complete... Image sparsify: 25% complete... Image sparsify: 26% complete... Image sparsify: 28% complete... Image sparsify: 29% complete... Image sparsify: 31% complete... Image sparsify: 32% complete... Image sparsify: 34% complete... Image sparsify: 35% complete... Image sparsify: 37% complete... Image sparsify: 39% complete... Image sparsify: 40% complete... Image sparsify: 42% complete... Image sparsify: 43% complete... Image sparsify: 45% complete... Image sparsify: 46% complete... Image sparsify: 48% complete... Image sparsify: 50% complete... Image sparsify: 51% complete... Image sparsify: 53% complete... Image sparsify: 54% complete... Image sparsify: 56% complete... Image sparsify: 57% complete... Image sparsify: 59% complete... Image sparsify: 60% complete... Image sparsify: 62% complete... Image sparsify: 64% complete... Image sparsify: 65% complete... Image sparsify: 67% complete... Image sparsify: 68% complete... Image sparsify: 70% complete... Image sparsify: 71% complete... Image sparsify: 73% complete... Image sparsify: 75% complete... Image sparsify: 76% complete... Image sparsify: 78% complete... Image sparsify: 79% complete... Image sparsify: 81% complete... Image sparsify: 82% complete... Image sparsify: 84% complete... Image sparsify: 85% complete... Image sparsify: 87% complete... Image sparsify: 89% complete... Image sparsify: 90% complete... Image sparsify: 92% complete... Image sparsify: 93% complete... Image sparsify: 95% complete... Image sparsify: 96% complete... Image sparsify: 98% complete... Image sparsify: 100% complete...done. 2023-12-24T18:00:45.413 INFO:tasks.workunit.client.0.smithi087.stderr:+ remove_images 2023-12-24T18:00:45.413 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:45.565 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:45.639 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:45.754 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:46.370 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:47.388 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:47.503 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:47.649 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:47.801 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:47.956 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:48.036 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:48.113 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:48.190 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:48.273 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:48.349 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:48.425 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:48.504 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:48.582 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:48.660 INFO:tasks.workunit.client.0.smithi087.stderr:+ rm -f /tmp/img1 /tmp/img1.new /tmp/img2 /tmp/img2.new /tmp/img3 /tmp/img3.new /tmp/img-diff1.new /tmp/img-diff2.new /tmp/img-diff3.new /tmp/img1.snap1 /tmp/img1.snap1 /tmp/img-diff1.snap1 2023-12-24T18:00:48.768 INFO:tasks.workunit.client.0.smithi087.stderr:+ test_locking 2023-12-24T18:00:48.768 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'testing locking...' 2023-12-24T18:00:48.768 INFO:tasks.workunit.client.0.smithi087.stdout:testing locking... 2023-12-24T18:00:48.769 INFO:tasks.workunit.client.0.smithi087.stderr:+ remove_images 2023-12-24T18:00:48.769 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:48.846 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:48.922 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:48.997 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:49.076 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:49.153 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:49.229 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:49.306 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:49.383 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:49.460 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:49.537 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:49.613 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:49.689 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:49.766 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:49.842 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:49.918 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:49.995 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:50.071 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:50.149 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 -s 1 test1 2023-12-24T18:00:50.195 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd lock list test1 2023-12-24T18:00:50.195 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:00:50.195 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^0$' 2023-12-24T18:00:50.230 INFO:tasks.workunit.client.0.smithi087.stdout:0 2023-12-24T18:00:50.230 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd lock add test1 id 2023-12-24T18:00:50.270 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd lock list test1 2023-12-24T18:00:50.270 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep ' 1 ' 2023-12-24T18:00:50.303 INFO:tasks.workunit.client.0.smithi087.stdout:There is 1 exclusive lock on this image. 2023-12-24T18:00:50.303 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd lock list test1 2023-12-24T18:00:50.304 INFO:tasks.workunit.client.0.smithi087.stderr:++ tail -n 1 2023-12-24T18:00:50.304 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1;}' 2023-12-24T18:00:50.337 INFO:tasks.workunit.client.0.smithi087.stderr:+ LOCKER=client.8462 2023-12-24T18:00:50.338 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd lock remove test1 id client.8462 2023-12-24T18:00:50.433 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd lock list test1 2023-12-24T18:00:50.434 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:00:50.434 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^0$' 2023-12-24T18:00:50.470 INFO:tasks.workunit.client.0.smithi087.stdout:0 2023-12-24T18:00:50.470 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd lock add test1 id --shared tag 2023-12-24T18:00:50.513 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd lock list test1 2023-12-24T18:00:50.513 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep ' 1 ' 2023-12-24T18:00:50.547 INFO:tasks.workunit.client.0.smithi087.stdout:There is 1 shared lock on this image. 2023-12-24T18:00:50.547 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd lock add test1 id --shared tag 2023-12-24T18:00:50.589 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd lock list test1 2023-12-24T18:00:50.589 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep ' 2 ' 2023-12-24T18:00:50.622 INFO:tasks.workunit.client.0.smithi087.stdout:There are 2 shared locks on this image. 2023-12-24T18:00:50.622 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd lock add test1 id2 --shared tag 2023-12-24T18:00:50.664 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd lock list test1 2023-12-24T18:00:50.664 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep ' 3 ' 2023-12-24T18:00:50.698 INFO:tasks.workunit.client.0.smithi087.stdout:There are 3 shared locks on this image. 2023-12-24T18:00:50.699 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd lock list test1 2023-12-24T18:00:50.699 INFO:tasks.workunit.client.0.smithi087.stderr:+ tail -n 1 2023-12-24T18:00:50.699 INFO:tasks.workunit.client.0.smithi087.stderr:+ awk '{print $2, $1;}' 2023-12-24T18:00:50.699 INFO:tasks.workunit.client.0.smithi087.stderr:+ xargs rbd lock remove test1 2023-12-24T18:00:51.445 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info test1 2023-12-24T18:00:51.445 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -qE 'features:.*exclusive' 2023-12-24T18:00:51.480 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd lock list test1 2023-12-24T18:00:51.515 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -n 'There are 2 shared locks on this image. 2023-12-24T18:00:51.516 INFO:tasks.workunit.client.0.smithi087.stderr:Lock tag: tag 2023-12-24T18:00:51.516 INFO:tasks.workunit.client.0.smithi087.stderr:Locker ID Address 2023-12-24T18:00:51.516 INFO:tasks.workunit.client.0.smithi087.stderr:client.8477 id 172.21.15.87:0/2563186987 2023-12-24T18:00:51.516 INFO:tasks.workunit.client.0.smithi087.stderr:client.8483 id 172.21.15.87:0/3357309370' ']' 2023-12-24T18:00:51.516 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd lock list test1 2023-12-24T18:00:51.516 INFO:tasks.workunit.client.0.smithi087.stderr:+ tail -n 1 2023-12-24T18:00:51.516 INFO:tasks.workunit.client.0.smithi087.stderr:+ awk '{print $2, $1;}' 2023-12-24T18:00:51.516 INFO:tasks.workunit.client.0.smithi087.stderr:+ xargs rbd lock remove test1 2023-12-24T18:00:52.072 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd lock list test1 2023-12-24T18:00:52.113 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -n 'There is 1 shared lock on this image. 2023-12-24T18:00:52.113 INFO:tasks.workunit.client.0.smithi087.stderr:Lock tag: tag 2023-12-24T18:00:52.113 INFO:tasks.workunit.client.0.smithi087.stderr:Locker ID Address 2023-12-24T18:00:52.113 INFO:tasks.workunit.client.0.smithi087.stderr:client.8477 id 172.21.15.87:0/2563186987' ']' 2023-12-24T18:00:52.113 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd lock list test1 2023-12-24T18:00:52.113 INFO:tasks.workunit.client.0.smithi087.stderr:+ tail -n 1 2023-12-24T18:00:52.113 INFO:tasks.workunit.client.0.smithi087.stderr:+ awk '{print $2, $1;}' 2023-12-24T18:00:52.113 INFO:tasks.workunit.client.0.smithi087.stderr:+ xargs rbd lock remove test1 2023-12-24T18:00:53.075 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd lock list test1 2023-12-24T18:00:53.113 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -n '' ']' 2023-12-24T18:00:53.113 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test1 2023-12-24T18:00:53.215 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:00:53.219+0000 7fe2345c0700 0 -- 172.21.15.87:0/1714224115 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x557e6cd5eb50 msgr2=0x557e6ce05da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:00:53.217 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T18:00:53.220 INFO:tasks.workunit.client.0.smithi087.stderr:+ test_clone 2023-12-24T18:00:53.220 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'testing clone...' 2023-12-24T18:00:53.221 INFO:tasks.workunit.client.0.smithi087.stdout:testing clone... 2023-12-24T18:00:53.221 INFO:tasks.workunit.client.0.smithi087.stderr:+ remove_images 2023-12-24T18:00:53.221 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:53.297 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:53.374 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:53.450 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:53.526 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:53.805 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:53.880 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:53.957 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:54.035 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:54.110 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:54.189 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:54.265 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:54.342 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:54.420 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:54.495 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:54.571 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:54.648 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:54.725 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:00:54.801 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create test1 --image-format 2 -s 1 2023-12-24T18:00:54.844 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create test1@s1 2023-12-24T18:00:55.076 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T18:00:55.084 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap protect test1@s1 2023-12-24T18:00:55.125 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph osd pool create rbd2 8 2023-12-24T18:00:56.199 INFO:tasks.workunit.client.0.smithi087.stderr:pool 'rbd2' already exists 2023-12-24T18:00:56.208 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd pool init rbd2 2023-12-24T18:00:59.086 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd clone test1@s1 rbd2/clone 2023-12-24T18:00:59.150 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd -p rbd2 ls 2023-12-24T18:00:59.151 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep clone 2023-12-24T18:00:59.381 INFO:tasks.workunit.client.0.smithi087.stdout:clone 2023-12-24T18:00:59.382 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd -p rbd2 ls -l 2023-12-24T18:00:59.382 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep clone 2023-12-24T18:00:59.382 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep test1@s1 2023-12-24T18:00:59.426 INFO:tasks.workunit.client.0.smithi087.stdout:clone 1 MiB rbd/test1@s1 2 2023-12-24T18:00:59.427 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T18:00:59.427 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -v clone 2023-12-24T18:00:59.454 INFO:tasks.workunit.client.0.smithi087.stdout:test1 2023-12-24T18:00:59.455 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd flatten rbd2/clone 2023-12-24T18:00:59.497 INFO:tasks.workunit.client.0.smithi087.stderr: Image flatten: 100% complete...done. 2023-12-24T18:00:59.502 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create rbd2/clone@s1 2023-12-24T18:01:00.089 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T18:01:00.095 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap protect rbd2/clone@s1 2023-12-24T18:01:00.135 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd clone rbd2/clone@s1 clone2 2023-12-24T18:01:00.208 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T18:01:00.208 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep clone2 2023-12-24T18:01:00.234 INFO:tasks.workunit.client.0.smithi087.stdout:clone2 2023-12-24T18:01:00.235 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls -l 2023-12-24T18:01:00.235 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep clone2 2023-12-24T18:01:00.235 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep rbd2/clone@s1 2023-12-24T18:01:00.275 INFO:tasks.workunit.client.0.smithi087.stdout:clone2 1 MiB rbd2/clone@s1 2 2023-12-24T18:01:00.276 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd -p rbd2 ls 2023-12-24T18:01:00.276 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -v clone2 2023-12-24T18:01:00.304 INFO:tasks.workunit.client.0.smithi087.stdout:clone 2023-12-24T18:01:00.304 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm clone2 2023-12-24T18:01:00.401 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:01:00.403+0000 7f2823a9c700 0 -- 172.21.15.87:0/4257503027 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f2800006530 msgr2=0x7f28000148c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:01:00.404 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T18:01:00.407 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap unprotect rbd2/clone@s1 2023-12-24T18:01:00.446 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap rm rbd2/clone@s1 2023-12-24T18:01:01.091 INFO:tasks.workunit.client.0.smithi087.stderr: Removing snap: 100% complete...done. 2023-12-24T18:01:01.100 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm rbd2/clone 2023-12-24T18:01:01.174 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:01:01.177+0000 7fb2b7fff700 0 -- 172.21.15.87:0/3667805263 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7fb298008740 msgr2=0x7fb29800abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:01:01.178 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T18:01:01.180 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap unprotect test1@s1 2023-12-24T18:01:01.223 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap rm test1@s1 2023-12-24T18:01:02.072 INFO:tasks.workunit.client.0.smithi087.stderr: Removing snap: 100% complete...done. 2023-12-24T18:01:02.084 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test1 2023-12-24T18:01:02.190 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:01:02.192+0000 7f7add900700 0 -- 172.21.15.87:0/2769506005 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f7ac40064f0 msgr2=0x7f7ac4011eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:01:02.190 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T18:01:02.193 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph osd pool rm rbd2 rbd2 --yes-i-really-really-mean-it 2023-12-24T18:01:03.215 INFO:tasks.workunit.client.0.smithi087.stderr:pool 'rbd2' does not exist 2023-12-24T18:01:03.224 INFO:tasks.workunit.client.0.smithi087.stderr:+ test_trash 2023-12-24T18:01:03.224 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'testing trash...' 2023-12-24T18:01:03.225 INFO:tasks.workunit.client.0.smithi087.stdout:testing trash... 2023-12-24T18:01:03.225 INFO:tasks.workunit.client.0.smithi087.stderr:+ remove_images 2023-12-24T18:01:03.226 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:03.307 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:03.384 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:03.461 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:03.538 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:03.615 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:03.691 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:03.767 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:03.845 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:03.920 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:04.198 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:04.274 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:04.352 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:04.427 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:04.504 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:04.580 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:04.655 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:04.732 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:04.809 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 -s 1 test1 2023-12-24T18:01:04.852 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 -s 1 test2 2023-12-24T18:01:04.897 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T18:01:04.897 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep test1 2023-12-24T18:01:04.924 INFO:tasks.workunit.client.0.smithi087.stdout:test1 2023-12-24T18:01:04.925 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T18:01:04.925 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep test2 2023-12-24T18:01:04.951 INFO:tasks.workunit.client.0.smithi087.stdout:test2 2023-12-24T18:01:04.951 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T18:01:04.951 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:04.952 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 2 2023-12-24T18:01:04.978 INFO:tasks.workunit.client.0.smithi087.stdout:2 2023-12-24T18:01:04.978 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls -l 2023-12-24T18:01:04.978 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'test1.*2.*' 2023-12-24T18:01:05.014 INFO:tasks.workunit.client.0.smithi087.stdout:test1 1 MiB 2 2023-12-24T18:01:05.015 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls -l 2023-12-24T18:01:05.015 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'test2.*2.*' 2023-12-24T18:01:05.050 INFO:tasks.workunit.client.0.smithi087.stdout:test2 1 MiB 2 2023-12-24T18:01:05.050 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv test1 2023-12-24T18:01:05.106 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T18:01:05.106 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep test2 2023-12-24T18:01:05.132 INFO:tasks.workunit.client.0.smithi087.stdout:test2 2023-12-24T18:01:05.133 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T18:01:05.133 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:05.133 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 1 2023-12-24T18:01:05.159 INFO:tasks.workunit.client.0.smithi087.stdout:1 2023-12-24T18:01:05.160 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls -l 2023-12-24T18:01:05.160 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'test2.*2.*' 2023-12-24T18:01:05.193 INFO:tasks.workunit.client.0.smithi087.stdout:test2 1 MiB 2 2023-12-24T18:01:05.193 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:05.194 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep test1 2023-12-24T18:01:05.220 INFO:tasks.workunit.client.0.smithi087.stdout:226f71de050f test1 2023-12-24T18:01:05.221 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:05.221 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:05.221 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 1 2023-12-24T18:01:05.248 INFO:tasks.workunit.client.0.smithi087.stdout:1 2023-12-24T18:01:05.248 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls -l 2023-12-24T18:01:05.248 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'test1.*USER.*' 2023-12-24T18:01:05.282 INFO:tasks.workunit.client.0.smithi087.stdout:226f71de050f test1 USER Sun Dec 24 18:01:05 2023 expired at Sun Dec 24 18:01:05 2023 2023-12-24T18:01:05.285 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls -l 2023-12-24T18:01:05.285 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -v 'protected until' 2023-12-24T18:01:05.316 INFO:tasks.workunit.client.0.smithi087.stdout:ID NAME SOURCE DELETED_AT STATUS PARENT 2023-12-24T18:01:05.317 INFO:tasks.workunit.client.0.smithi087.stdout:226f71de050f test1 USER Sun Dec 24 18:01:05 2023 expired at Sun Dec 24 18:01:05 2023 2023-12-24T18:01:05.317 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd trash ls 2023-12-24T18:01:05.317 INFO:tasks.workunit.client.0.smithi087.stderr:++ cut -d ' ' -f 1 2023-12-24T18:01:05.343 INFO:tasks.workunit.client.0.smithi087.stderr:+ ID=226f71de050f 2023-12-24T18:01:05.344 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash rm 226f71de050f 2023-12-24T18:01:05.398 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T18:01:05.401 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv test2 2023-12-24T18:01:05.458 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd trash ls 2023-12-24T18:01:05.459 INFO:tasks.workunit.client.0.smithi087.stderr:++ cut -d ' ' -f 1 2023-12-24T18:01:05.487 INFO:tasks.workunit.client.0.smithi087.stderr:+ ID=22722851601e 2023-12-24T18:01:05.488 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info --image-id 22722851601e 2023-12-24T18:01:05.488 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'rbd image '\''test2'\''' 2023-12-24T18:01:05.519 INFO:tasks.workunit.client.0.smithi087.stdout:rbd image 'test2': 2023-12-24T18:01:05.520 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash restore 22722851601e 2023-12-24T18:01:05.556 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T18:01:05.556 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep test2 2023-12-24T18:01:05.583 INFO:tasks.workunit.client.0.smithi087.stdout:test2 2023-12-24T18:01:05.583 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T18:01:05.583 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:05.583 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 1 2023-12-24T18:01:05.610 INFO:tasks.workunit.client.0.smithi087.stdout:1 2023-12-24T18:01:05.611 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls -l 2023-12-24T18:01:05.611 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'test2.*2.*' 2023-12-24T18:01:05.644 INFO:tasks.workunit.client.0.smithi087.stdout:test2 1 MiB 2 2023-12-24T18:01:05.644 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv test2 --expires-at '3600 sec' 2023-12-24T18:01:05.698 INFO:tasks.workunit.client.0.smithi087.stdout:rbd: image test2 will expire at 2023-12-24T19:01:05.676142+0000 2023-12-24T18:01:05.701 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:05.701 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep test2 2023-12-24T18:01:05.728 INFO:tasks.workunit.client.0.smithi087.stdout:22722851601e test2 2023-12-24T18:01:05.729 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:05.729 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:05.729 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 1 2023-12-24T18:01:05.755 INFO:tasks.workunit.client.0.smithi087.stdout:1 2023-12-24T18:01:05.755 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls -l 2023-12-24T18:01:05.755 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'test2.*USER.*protected until' 2023-12-24T18:01:05.789 INFO:tasks.workunit.client.0.smithi087.stdout:22722851601e test2 USER Sun Dec 24 18:01:05 2023 protected until Sun Dec 24 19:01:05 2023 2023-12-24T18:01:05.790 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash rm 22722851601e 2023-12-24T18:01:05.790 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'Deferment time has not expired' 2023-12-24T18:01:05.818 INFO:tasks.workunit.client.0.smithi087.stdout:Deferment time has not expired, please use --force if you really want to remove the image 2023-12-24T18:01:05.818 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash rm --image-id 22722851601e --force 2023-12-24T18:01:05.879 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T18:01:05.881 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 -s 1 test1 2023-12-24T18:01:05.924 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create test1@snap1 2023-12-24T18:01:06.106 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T18:01:06.114 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap protect test1@snap1 2023-12-24T18:01:06.154 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv test1 2023-12-24T18:01:06.209 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:06.209 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep test1 2023-12-24T18:01:06.236 INFO:tasks.workunit.client.0.smithi087.stdout:22c99af21b51 test1 2023-12-24T18:01:06.237 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:06.237 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:06.237 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 1 2023-12-24T18:01:06.263 INFO:tasks.workunit.client.0.smithi087.stdout:1 2023-12-24T18:01:06.264 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls -l 2023-12-24T18:01:06.264 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'test1.*USER.*' 2023-12-24T18:01:06.297 INFO:tasks.workunit.client.0.smithi087.stdout:22c99af21b51 test1 USER Sun Dec 24 18:01:06 2023 expired at Sun Dec 24 18:01:06 2023 2023-12-24T18:01:06.298 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls -l 2023-12-24T18:01:06.298 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -v 'protected until' 2023-12-24T18:01:06.332 INFO:tasks.workunit.client.0.smithi087.stdout:ID NAME SOURCE DELETED_AT STATUS PARENT 2023-12-24T18:01:06.332 INFO:tasks.workunit.client.0.smithi087.stdout:22c99af21b51 test1 USER Sun Dec 24 18:01:06 2023 expired at Sun Dec 24 18:01:06 2023 2023-12-24T18:01:06.333 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd trash ls 2023-12-24T18:01:06.333 INFO:tasks.workunit.client.0.smithi087.stderr:++ cut -d ' ' -f 1 2023-12-24T18:01:06.360 INFO:tasks.workunit.client.0.smithi087.stderr:+ ID=22c99af21b51 2023-12-24T18:01:06.360 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap ls --image-id 22c99af21b51 2023-12-24T18:01:06.360 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -v SNAPID 2023-12-24T18:01:06.360 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:06.360 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 1 2023-12-24T18:01:06.395 INFO:tasks.workunit.client.0.smithi087.stdout:1 2023-12-24T18:01:06.395 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap ls --image-id 22c99af21b51 2023-12-24T18:01:06.395 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '.*snap1.*' 2023-12-24T18:01:06.430 INFO:tasks.workunit.client.0.smithi087.stdout: 34 snap1 1 MiB yes Sun Dec 24 18:01:06 2023 2023-12-24T18:01:06.430 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap unprotect --image-id 22c99af21b51 --snap snap1 2023-12-24T18:01:06.475 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap rm --image-id 22c99af21b51 --snap snap1 2023-12-24T18:01:07.102 INFO:tasks.workunit.client.0.smithi087.stderr: Removing snap: 100% complete...done. 2023-12-24T18:01:07.115 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap ls --image-id 22c99af21b51 2023-12-24T18:01:07.115 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -v SNAPID 2023-12-24T18:01:07.115 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:07.115 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 0 2023-12-24T18:01:07.151 INFO:tasks.workunit.client.0.smithi087.stdout:0 2023-12-24T18:01:07.152 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash restore 22c99af21b51 2023-12-24T18:01:07.190 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create test1@snap1 2023-12-24T18:01:08.113 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T18:01:08.124 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create test1@snap2 2023-12-24T18:01:09.126 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T18:01:09.137 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap ls --image-id 22c99af21b51 2023-12-24T18:01:09.138 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -v SNAPID 2023-12-24T18:01:09.138 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:09.138 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 2 2023-12-24T18:01:09.178 INFO:tasks.workunit.client.0.smithi087.stdout:2 2023-12-24T18:01:09.178 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap purge --image-id 22c99af21b51 2023-12-24T18:01:11.127 INFO:tasks.workunit.client.0.smithi087.stderr: Removing all snapshots: 50% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2023-12-24T18:01:11.139 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap ls --image-id 22c99af21b51 2023-12-24T18:01:11.139 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -v SNAPID 2023-12-24T18:01:11.139 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:11.139 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 0 2023-12-24T18:01:11.177 INFO:tasks.workunit.client.0.smithi087.stdout:0 2023-12-24T18:01:11.178 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm --rbd_move_to_trash_on_remove=true --rbd_move_to_trash_on_remove_expire_seconds=3600 test1 2023-12-24T18:01:11.234 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T18:01:11.237 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:11.238 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep test1 2023-12-24T18:01:11.264 INFO:tasks.workunit.client.0.smithi087.stdout:22c99af21b51 test1 2023-12-24T18:01:11.265 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:11.265 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:11.265 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 1 2023-12-24T18:01:11.291 INFO:tasks.workunit.client.0.smithi087.stdout:1 2023-12-24T18:01:11.291 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls -l 2023-12-24T18:01:11.291 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'test1.*USER.*protected until' 2023-12-24T18:01:11.325 INFO:tasks.workunit.client.0.smithi087.stdout:22c99af21b51 test1 USER Sun Dec 24 18:01:11 2023 protected until Sun Dec 24 19:01:11 2023 2023-12-24T18:01:11.325 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash rm 22c99af21b51 2023-12-24T18:01:11.325 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'Deferment time has not expired' 2023-12-24T18:01:11.353 INFO:tasks.workunit.client.0.smithi087.stdout:Deferment time has not expired, please use --force if you really want to remove the image 2023-12-24T18:01:11.353 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash rm --image-id 22c99af21b51 --force 2023-12-24T18:01:11.413 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T18:01:11.415 INFO:tasks.workunit.client.0.smithi087.stderr:+ remove_images 2023-12-24T18:01:11.415 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:11.493 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:11.573 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:11.648 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:11.725 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:11.803 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:11.880 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:11.957 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:12.034 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:12.116 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:12.194 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:12.271 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:12.349 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:12.427 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:12.505 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:12.581 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:12.860 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:12.938 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:13.016 INFO:tasks.workunit.client.0.smithi087.stderr:+ test_purge 2023-12-24T18:01:13.016 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'testing trash purge...' 2023-12-24T18:01:13.017 INFO:tasks.workunit.client.0.smithi087.stdout:testing trash purge... 2023-12-24T18:01:13.017 INFO:tasks.workunit.client.0.smithi087.stderr:+ remove_images 2023-12-24T18:01:13.017 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:13.095 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:13.171 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:13.249 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:13.326 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:13.403 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:13.480 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:13.558 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:13.636 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:13.712 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:13.790 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:13.867 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:13.944 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:14.020 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:14.099 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:14.176 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:14.253 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:14.330 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:14.408 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:14.408 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:14.408 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 0 2023-12-24T18:01:14.435 INFO:tasks.workunit.client.0.smithi087.stdout:0 2023-12-24T18:01:14.435 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash purge 2023-12-24T18:01:14.460 INFO:tasks.workunit.client.0.smithi087.stderr: Removing images: 100% complete...done. 2023-12-24T18:01:14.462 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 --size 256 testimg1 2023-12-24T18:01:14.505 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 --size 256 testimg2 2023-12-24T18:01:14.548 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv testimg1 2023-12-24T18:01:14.604 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv testimg2 2023-12-24T18:01:14.662 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:14.662 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:14.662 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 2 2023-12-24T18:01:14.688 INFO:tasks.workunit.client.0.smithi087.stdout:2 2023-12-24T18:01:14.688 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash purge 2023-12-24T18:01:14.776 INFO:tasks.workunit.client.0.smithi087.stderr: Removing images: 50% complete... Removing images: 100% complete... Removing images: 100% complete...done. 2023-12-24T18:01:14.779 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:14.779 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:14.779 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 0 2023-12-24T18:01:14.806 INFO:tasks.workunit.client.0.smithi087.stdout:0 2023-12-24T18:01:14.806 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 --size 256 testimg1 2023-12-24T18:01:14.848 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 --size 256 testimg2 2023-12-24T18:01:14.891 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv testimg1 --expires-at '1 hour' 2023-12-24T18:01:14.946 INFO:tasks.workunit.client.0.smithi087.stdout:rbd: image testimg1 will expire at 2023-12-24T19:01:14.924053+0000 2023-12-24T18:01:14.948 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv testimg2 --expires-at '3 hours' 2023-12-24T18:01:15.003 INFO:tasks.workunit.client.0.smithi087.stdout:rbd: image testimg2 will expire at 2023-12-24T21:01:14.981396+0000 2023-12-24T18:01:15.006 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:15.006 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:15.006 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 2 2023-12-24T18:01:15.033 INFO:tasks.workunit.client.0.smithi087.stdout:2 2023-12-24T18:01:15.033 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash purge 2023-12-24T18:01:15.058 INFO:tasks.workunit.client.0.smithi087.stderr: Removing images: 100% complete...done. 2023-12-24T18:01:15.060 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:15.060 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:15.060 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 2 2023-12-24T18:01:15.087 INFO:tasks.workunit.client.0.smithi087.stdout:2 2023-12-24T18:01:15.087 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash purge --expired-before 'now + 2 hours' 2023-12-24T18:01:15.148 INFO:tasks.workunit.client.0.smithi087.stderr: Removing images: 100% complete... Removing images: 100% complete...done. 2023-12-24T18:01:15.151 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:15.151 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:15.151 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 1 2023-12-24T18:01:15.178 INFO:tasks.workunit.client.0.smithi087.stdout:1 2023-12-24T18:01:15.178 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:15.178 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep testimg2 2023-12-24T18:01:15.405 INFO:tasks.workunit.client.0.smithi087.stdout:240d400561ea testimg2 2023-12-24T18:01:15.405 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash purge --expired-before 'now + 4 hours' 2023-12-24T18:01:15.467 INFO:tasks.workunit.client.0.smithi087.stderr: Removing images: 100% complete... Removing images: 100% complete...done. 2023-12-24T18:01:15.470 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:15.470 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:15.470 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 0 2023-12-24T18:01:15.497 INFO:tasks.workunit.client.0.smithi087.stdout:0 2023-12-24T18:01:15.497 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 --size 256 testimg1 2023-12-24T18:01:15.544 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create testimg1@snap 2023-12-24T18:01:16.134 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T18:01:16.143 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 --size 256 testimg2 2023-12-24T18:01:16.186 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 --size 256 testimg3 2023-12-24T18:01:16.230 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv testimg1 2023-12-24T18:01:16.287 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv testimg2 2023-12-24T18:01:16.347 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv testimg3 2023-12-24T18:01:16.406 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:16.406 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:16.406 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 3 2023-12-24T18:01:16.432 INFO:tasks.workunit.client.0.smithi087.stdout:3 2023-12-24T18:01:16.433 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash purge 2023-12-24T18:01:16.433 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'some expired images could not be removed' 2023-12-24T18:01:16.562 INFO:tasks.workunit.client.0.smithi087.stdout:rbd: some expired images could not be removed 2023-12-24T18:01:16.563 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:16.563 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:16.563 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 1 2023-12-24T18:01:16.589 INFO:tasks.workunit.client.0.smithi087.stdout:1 2023-12-24T18:01:16.590 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:16.590 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep testimg1 2023-12-24T18:01:16.616 INFO:tasks.workunit.client.0.smithi087.stdout:242ee3900d38 testimg1 2023-12-24T18:01:16.616 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd trash ls 2023-12-24T18:01:16.616 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{ print $1 }' 2023-12-24T18:01:16.644 INFO:tasks.workunit.client.0.smithi087.stderr:+ ID=242ee3900d38 2023-12-24T18:01:16.644 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap purge --image-id 242ee3900d38 2023-12-24T18:01:17.131 INFO:tasks.workunit.client.0.smithi087.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2023-12-24T18:01:17.142 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash purge 2023-12-24T18:01:17.200 INFO:tasks.workunit.client.0.smithi087.stderr: Removing images: 100% complete... Removing images: 100% complete...done. 2023-12-24T18:01:17.203 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:17.203 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:17.203 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 0 2023-12-24T18:01:17.229 INFO:tasks.workunit.client.0.smithi087.stdout:0 2023-12-24T18:01:17.230 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 --size 256 testimg1 2023-12-24T18:01:17.273 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 --size 256 testimg2 2023-12-24T18:01:17.318 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create testimg2@snap 2023-12-24T18:01:18.148 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T18:01:18.158 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 --size 256 testimg3 2023-12-24T18:01:18.206 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv testimg1 2023-12-24T18:01:18.280 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv testimg2 2023-12-24T18:01:18.336 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv testimg3 2023-12-24T18:01:18.393 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:18.393 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:18.393 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 3 2023-12-24T18:01:18.419 INFO:tasks.workunit.client.0.smithi087.stdout:3 2023-12-24T18:01:18.419 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash purge 2023-12-24T18:01:18.420 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'some expired images could not be removed' 2023-12-24T18:01:18.558 INFO:tasks.workunit.client.0.smithi087.stdout:rbd: some expired images could not be removed 2023-12-24T18:01:18.558 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:18.558 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:18.558 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 1 2023-12-24T18:01:18.585 INFO:tasks.workunit.client.0.smithi087.stdout:1 2023-12-24T18:01:18.585 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:18.586 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep testimg2 2023-12-24T18:01:18.813 INFO:tasks.workunit.client.0.smithi087.stdout:245ec4ff6bf7 testimg2 2023-12-24T18:01:18.813 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd trash ls 2023-12-24T18:01:18.813 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{ print $1 }' 2023-12-24T18:01:18.841 INFO:tasks.workunit.client.0.smithi087.stderr:+ ID=245ec4ff6bf7 2023-12-24T18:01:18.841 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap purge --image-id 245ec4ff6bf7 2023-12-24T18:01:19.144 INFO:tasks.workunit.client.0.smithi087.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2023-12-24T18:01:19.156 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash purge 2023-12-24T18:01:19.212 INFO:tasks.workunit.client.0.smithi087.stderr: Removing images: 100% complete... Removing images: 100% complete...done. 2023-12-24T18:01:19.215 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:19.215 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:19.215 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 0 2023-12-24T18:01:19.242 INFO:tasks.workunit.client.0.smithi087.stdout:0 2023-12-24T18:01:19.242 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 --size 256 testimg1 2023-12-24T18:01:19.285 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 --size 256 testimg2 2023-12-24T18:01:19.328 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 --size 256 testimg3 2023-12-24T18:01:19.370 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create testimg3@snap 2023-12-24T18:01:20.161 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T18:01:20.169 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv testimg1 2023-12-24T18:01:20.230 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv testimg2 2023-12-24T18:01:20.292 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv testimg3 2023-12-24T18:01:20.347 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:20.348 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:20.348 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 3 2023-12-24T18:01:20.374 INFO:tasks.workunit.client.0.smithi087.stdout:3 2023-12-24T18:01:20.374 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash purge 2023-12-24T18:01:20.374 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'some expired images could not be removed' 2023-12-24T18:01:20.711 INFO:tasks.workunit.client.0.smithi087.stdout:rbd: some expired images could not be removed 2023-12-24T18:01:20.712 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:20.712 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:20.712 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 1 2023-12-24T18:01:20.738 INFO:tasks.workunit.client.0.smithi087.stdout:1 2023-12-24T18:01:20.738 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:20.738 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep testimg3 2023-12-24T18:01:20.959 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:01:20.962+0000 7f2f4feac700 0 --2- 172.21.15.87:0/205752505 >> v2:172.21.15.87:3300/0 conn(0x56461b15f8c0 0x56461b163500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-12-24T18:01:20.966 INFO:tasks.workunit.client.0.smithi087.stdout:248ea34a683d testimg3 2023-12-24T18:01:20.967 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd trash ls 2023-12-24T18:01:20.967 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{ print $1 }' 2023-12-24T18:01:20.995 INFO:tasks.workunit.client.0.smithi087.stderr:+ ID=248ea34a683d 2023-12-24T18:01:20.995 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap purge --image-id 248ea34a683d 2023-12-24T18:01:21.153 INFO:tasks.workunit.client.0.smithi087.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2023-12-24T18:01:21.162 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash purge 2023-12-24T18:01:21.218 INFO:tasks.workunit.client.0.smithi087.stderr: Removing images: 100% complete... Removing images: 100% complete...done. 2023-12-24T18:01:21.221 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:21.221 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:21.221 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 0 2023-12-24T18:01:21.247 INFO:tasks.workunit.client.0.smithi087.stdout:0 2023-12-24T18:01:21.247 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 --size 256 testimg1 2023-12-24T18:01:21.291 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create testimg1@snap 2023-12-24T18:01:22.084 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T18:01:22.094 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd clone --rbd-default-clone-format=2 testimg1@snap testimg2 2023-12-24T18:01:22.165 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap rm testimg1@snap 2023-12-24T18:01:22.208 INFO:tasks.workunit.client.0.smithi087.stderr: Removing snap: 100% complete...done. 2023-12-24T18:01:22.216 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 --size 256 testimg3 2023-12-24T18:01:22.260 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create testimg2@snap 2023-12-24T18:01:23.170 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T18:01:23.180 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd clone --rbd-default-clone-format=2 testimg2@snap testimg4 2023-12-24T18:01:23.257 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd clone --rbd-default-clone-format=2 testimg2@snap testimg5 2023-12-24T18:01:23.336 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap rm testimg2@snap 2023-12-24T18:01:23.382 INFO:tasks.workunit.client.0.smithi087.stderr: Removing snap: 100% complete...done. 2023-12-24T18:01:23.389 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create testimg4@snap 2023-12-24T18:01:24.169 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T18:01:24.179 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd clone --rbd-default-clone-format=2 testimg4@snap testimg6 2023-12-24T18:01:24.263 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap rm testimg4@snap 2023-12-24T18:01:24.317 INFO:tasks.workunit.client.0.smithi087.stderr: Removing snap: 100% complete...done. 2023-12-24T18:01:24.325 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv testimg1 2023-12-24T18:01:24.380 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv testimg2 2023-12-24T18:01:24.435 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv testimg3 2023-12-24T18:01:24.495 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv testimg4 2023-12-24T18:01:24.558 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:24.558 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:24.558 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 4 2023-12-24T18:01:24.585 INFO:tasks.workunit.client.0.smithi087.stdout:4 2023-12-24T18:01:24.585 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash purge 2023-12-24T18:01:24.586 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'some expired images could not be removed' 2023-12-24T18:01:24.775 INFO:tasks.workunit.client.0.smithi087.stdout:rbd: some expired images could not be removed 2023-12-24T18:01:24.775 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:24.776 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:24.776 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 3 2023-12-24T18:01:24.802 INFO:tasks.workunit.client.0.smithi087.stdout:3 2023-12-24T18:01:24.803 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:24.803 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep testimg1 2023-12-24T18:01:24.829 INFO:tasks.workunit.client.0.smithi087.stdout:24b581d196e5 testimg1 2023-12-24T18:01:24.830 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:24.830 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep testimg2 2023-12-24T18:01:24.857 INFO:tasks.workunit.client.0.smithi087.stdout:24bbeb9386c5 testimg2 2023-12-24T18:01:24.857 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:24.857 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep testimg4 2023-12-24T18:01:24.883 INFO:tasks.workunit.client.0.smithi087.stdout:24c7441ddc70 testimg4 2023-12-24T18:01:24.884 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv testimg6 2023-12-24T18:01:24.939 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:24.939 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:24.939 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 4 2023-12-24T18:01:24.966 INFO:tasks.workunit.client.0.smithi087.stdout:4 2023-12-24T18:01:24.966 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash purge 2023-12-24T18:01:24.966 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'some expired images could not be removed' 2023-12-24T18:01:25.319 INFO:tasks.workunit.client.0.smithi087.stdout:rbd: some expired images could not be removed 2023-12-24T18:01:25.320 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:25.320 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:25.320 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 2 2023-12-24T18:01:25.346 INFO:tasks.workunit.client.0.smithi087.stdout:2 2023-12-24T18:01:25.346 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:25.347 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep testimg1 2023-12-24T18:01:25.373 INFO:tasks.workunit.client.0.smithi087.stdout:24b581d196e5 testimg1 2023-12-24T18:01:25.373 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:25.373 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep testimg2 2023-12-24T18:01:25.400 INFO:tasks.workunit.client.0.smithi087.stdout:24bbeb9386c5 testimg2 2023-12-24T18:01:25.401 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv testimg5 2023-12-24T18:01:25.457 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:25.457 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:25.457 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 3 2023-12-24T18:01:25.483 INFO:tasks.workunit.client.0.smithi087.stdout:3 2023-12-24T18:01:25.483 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash purge 2023-12-24T18:01:25.568 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:01:25.571+0000 7f6f0bae9700 0 -- 172.21.15.87:0/2929935177 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7f6eec054b30 msgr2=0x7f6eec056f10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:01:25.578 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:01:25.580+0000 7f6f0c2ea700 0 -- 172.21.15.87:0/2929935177 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x558db8cec970 msgr2=0x558db8d96ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:01:27.250 INFO:tasks.workunit.client.0.smithi087.stderr: Removing images: 33% complete... Removing images: 66% complete... Removing images: 100% complete... Removing images: 100% complete...done. 2023-12-24T18:01:27.252 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:27.252 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:27.252 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 0 2023-12-24T18:01:27.280 INFO:tasks.workunit.client.0.smithi087.stdout:0 2023-12-24T18:01:27.280 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 --size 256 testimg1 2023-12-24T18:01:27.324 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create testimg1@snap 2023-12-24T18:01:28.198 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T18:01:28.210 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd clone --rbd-default-clone-format=2 testimg1@snap testimg2 2023-12-24T18:01:28.280 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap rm testimg1@snap 2023-12-24T18:01:28.322 INFO:tasks.workunit.client.0.smithi087.stderr: Removing snap: 100% complete...done. 2023-12-24T18:01:28.331 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 --size 256 testimg3 2023-12-24T18:01:28.374 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create testimg3@snap 2023-12-24T18:01:29.200 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T18:01:29.210 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create testimg2@snap 2023-12-24T18:01:30.205 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T18:01:30.215 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd clone --rbd-default-clone-format=2 testimg2@snap testimg4 2023-12-24T18:01:30.302 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd clone --rbd-default-clone-format=2 testimg2@snap testimg5 2023-12-24T18:01:30.373 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap rm testimg2@snap 2023-12-24T18:01:30.422 INFO:tasks.workunit.client.0.smithi087.stderr: Removing snap: 100% complete...done. 2023-12-24T18:01:30.431 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create testimg4@snap 2023-12-24T18:01:31.217 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T18:01:31.227 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd clone --rbd-default-clone-format=2 testimg4@snap testimg6 2023-12-24T18:01:31.314 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap rm testimg4@snap 2023-12-24T18:01:31.369 INFO:tasks.workunit.client.0.smithi087.stderr: Removing snap: 100% complete...done. 2023-12-24T18:01:31.377 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv testimg1 2023-12-24T18:01:31.433 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv testimg2 2023-12-24T18:01:31.490 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv testimg3 2023-12-24T18:01:31.547 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv testimg4 2023-12-24T18:01:31.605 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:31.605 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:31.605 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 4 2023-12-24T18:01:31.631 INFO:tasks.workunit.client.0.smithi087.stdout:4 2023-12-24T18:01:31.632 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash purge 2023-12-24T18:01:31.632 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'some expired images could not be removed' 2023-12-24T18:01:31.754 INFO:tasks.workunit.client.0.smithi087.stdout:rbd: some expired images could not be removed 2023-12-24T18:01:31.754 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:31.755 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:31.755 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 4 2023-12-24T18:01:31.781 INFO:tasks.workunit.client.0.smithi087.stdout:4 2023-12-24T18:01:31.782 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv testimg6 2023-12-24T18:01:31.838 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:31.838 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:31.838 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 5 2023-12-24T18:01:31.864 INFO:tasks.workunit.client.0.smithi087.stdout:5 2023-12-24T18:01:31.865 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash purge 2023-12-24T18:01:31.865 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'some expired images could not be removed' 2023-12-24T18:01:32.284 INFO:tasks.workunit.client.0.smithi087.stdout:rbd: some expired images could not be removed 2023-12-24T18:01:32.284 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:32.284 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:32.284 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 3 2023-12-24T18:01:32.310 INFO:tasks.workunit.client.0.smithi087.stdout:3 2023-12-24T18:01:32.311 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:32.311 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep testimg1 2023-12-24T18:01:32.338 INFO:tasks.workunit.client.0.smithi087.stdout:2515ce264348 testimg1 2023-12-24T18:01:32.338 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:32.338 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep testimg2 2023-12-24T18:01:32.365 INFO:tasks.workunit.client.0.smithi087.stdout:251b710a405c testimg2 2023-12-24T18:01:32.366 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:32.366 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep testimg3 2023-12-24T18:01:32.393 INFO:tasks.workunit.client.0.smithi087.stdout:25218e1186c7 testimg3 2023-12-24T18:01:32.393 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv testimg5 2023-12-24T18:01:32.453 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:32.453 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:32.453 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 4 2023-12-24T18:01:32.479 INFO:tasks.workunit.client.0.smithi087.stdout:4 2023-12-24T18:01:32.479 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash purge 2023-12-24T18:01:32.480 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'some expired images could not be removed' 2023-12-24T18:01:34.356 INFO:tasks.workunit.client.0.smithi087.stdout:rbd: some expired images could not be removed 2023-12-24T18:01:34.357 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:34.357 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:34.357 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 1 2023-12-24T18:01:34.383 INFO:tasks.workunit.client.0.smithi087.stdout:1 2023-12-24T18:01:34.384 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:34.384 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep testimg3 2023-12-24T18:01:34.410 INFO:tasks.workunit.client.0.smithi087.stdout:25218e1186c7 testimg3 2023-12-24T18:01:34.411 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd trash ls 2023-12-24T18:01:34.411 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{ print $1 }' 2023-12-24T18:01:34.438 INFO:tasks.workunit.client.0.smithi087.stderr:+ ID=25218e1186c7 2023-12-24T18:01:34.438 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap purge --image-id 25218e1186c7 2023-12-24T18:01:35.214 INFO:tasks.workunit.client.0.smithi087.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2023-12-24T18:01:35.225 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash purge 2023-12-24T18:01:35.293 INFO:tasks.workunit.client.0.smithi087.stderr: Removing images: 100% complete... Removing images: 100% complete...done. 2023-12-24T18:01:35.296 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:35.296 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:35.296 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 0 2023-12-24T18:01:35.324 INFO:tasks.workunit.client.0.smithi087.stdout:0 2023-12-24T18:01:35.324 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 --size 256 testimg1 2023-12-24T18:01:35.373 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create testimg1@snap 2023-12-24T18:01:36.222 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T18:01:36.232 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd clone --rbd-default-clone-format=2 testimg1@snap testimg2 2023-12-24T18:01:36.299 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap rm testimg1@snap 2023-12-24T18:01:36.341 INFO:tasks.workunit.client.0.smithi087.stderr: Removing snap: 100% complete...done. 2023-12-24T18:01:36.349 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 --size 256 testimg3 2023-12-24T18:01:36.392 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create testimg2@snap 2023-12-24T18:01:37.224 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T18:01:37.234 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd clone --rbd-default-clone-format=2 testimg2@snap testimg4 2023-12-24T18:01:37.312 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd clone --rbd-default-clone-format=2 testimg2@snap testimg5 2023-12-24T18:01:37.386 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap rm testimg2@snap 2023-12-24T18:01:37.431 INFO:tasks.workunit.client.0.smithi087.stderr: Removing snap: 100% complete...done. 2023-12-24T18:01:37.439 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create testimg4@snap 2023-12-24T18:01:38.232 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T18:01:38.243 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd clone --rbd-default-clone-format=2 testimg4@snap testimg6 2023-12-24T18:01:38.326 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap rm testimg4@snap 2023-12-24T18:01:38.379 INFO:tasks.workunit.client.0.smithi087.stderr: Removing snap: 100% complete...done. 2023-12-24T18:01:38.388 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm --rbd_move_parent_to_trash_on_remove=true testimg1 2023-12-24T18:01:38.463 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T18:01:38.465 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm --rbd_move_parent_to_trash_on_remove=true testimg2 2023-12-24T18:01:38.536 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T18:01:38.538 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv testimg3 2023-12-24T18:01:38.603 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm --rbd_move_parent_to_trash_on_remove=true testimg4 2023-12-24T18:01:38.677 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T18:01:38.679 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:38.679 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:38.679 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 4 2023-12-24T18:01:38.706 INFO:tasks.workunit.client.0.smithi087.stdout:4 2023-12-24T18:01:38.706 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash purge 2023-12-24T18:01:38.706 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'some expired images could not be removed' 2023-12-24T18:01:38.903 INFO:tasks.workunit.client.0.smithi087.stdout:rbd: some expired images could not be removed 2023-12-24T18:01:38.904 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:38.904 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:38.904 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 3 2023-12-24T18:01:38.930 INFO:tasks.workunit.client.0.smithi087.stdout:3 2023-12-24T18:01:38.931 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:38.931 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep testimg1 2023-12-24T18:01:38.957 INFO:tasks.workunit.client.0.smithi087.stdout:25802c073901 testimg1 2023-12-24T18:01:38.957 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:38.957 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep testimg2 2023-12-24T18:01:38.984 INFO:tasks.workunit.client.0.smithi087.stdout:258644ef54cd testimg2 2023-12-24T18:01:38.984 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:38.984 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep testimg4 2023-12-24T18:01:39.011 INFO:tasks.workunit.client.0.smithi087.stdout:2592ad009419 testimg4 2023-12-24T18:01:39.011 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv testimg6 2023-12-24T18:01:39.068 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:39.068 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:39.068 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 4 2023-12-24T18:01:39.094 INFO:tasks.workunit.client.0.smithi087.stdout:4 2023-12-24T18:01:39.095 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash purge 2023-12-24T18:01:39.095 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'some expired images could not be removed' 2023-12-24T18:01:40.348 INFO:tasks.workunit.client.0.smithi087.stdout:rbd: some expired images could not be removed 2023-12-24T18:01:40.349 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:40.349 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:40.349 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 2 2023-12-24T18:01:40.375 INFO:tasks.workunit.client.0.smithi087.stdout:2 2023-12-24T18:01:40.376 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:40.376 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep testimg1 2023-12-24T18:01:40.402 INFO:tasks.workunit.client.0.smithi087.stdout:25802c073901 testimg1 2023-12-24T18:01:40.403 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:40.403 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep testimg2 2023-12-24T18:01:40.429 INFO:tasks.workunit.client.0.smithi087.stdout:258644ef54cd testimg2 2023-12-24T18:01:40.430 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv testimg5 2023-12-24T18:01:40.487 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:40.487 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:40.487 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 3 2023-12-24T18:01:40.513 INFO:tasks.workunit.client.0.smithi087.stdout:3 2023-12-24T18:01:40.513 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash purge 2023-12-24T18:01:40.601 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:01:40.605+0000 7f4b0382c700 0 -- 172.21.15.87:0/597596029 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7f4adc054b30 msgr2=0x7f4adc056f10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:01:40.612 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:01:40.616+0000 7f4b0402d700 0 -- 172.21.15.87:0/597596029 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x55fa2b2856b0 msgr2=0x55fa2b299db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-24T18:01:42.151 INFO:tasks.workunit.client.0.smithi087.stderr: Removing images: 33% complete... Removing images: 66% complete... Removing images: 100% complete... Removing images: 100% complete...done. 2023-12-24T18:01:42.154 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:42.154 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:42.154 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 0 2023-12-24T18:01:42.182 INFO:tasks.workunit.client.0.smithi087.stdout:0 2023-12-24T18:01:42.182 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 --size 256 testimg1 2023-12-24T18:01:42.229 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create testimg1@snap 2023-12-24T18:01:43.252 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T18:01:43.263 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd clone --rbd-default-clone-format=2 testimg1@snap testimg2 2023-12-24T18:01:43.540 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap rm testimg1@snap 2023-12-24T18:01:43.585 INFO:tasks.workunit.client.0.smithi087.stderr: Removing snap: 100% complete...done. 2023-12-24T18:01:43.594 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 --size 256 testimg3 2023-12-24T18:01:43.641 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create testimg3@snap 2023-12-24T18:01:44.254 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T18:01:44.263 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create testimg2@snap 2023-12-24T18:01:45.257 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T18:01:45.268 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd clone --rbd-default-clone-format=2 testimg2@snap testimg4 2023-12-24T18:01:45.346 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd clone --rbd-default-clone-format=2 testimg2@snap testimg5 2023-12-24T18:01:45.424 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap rm testimg2@snap 2023-12-24T18:01:45.469 INFO:tasks.workunit.client.0.smithi087.stderr: Removing snap: 100% complete...done. 2023-12-24T18:01:45.477 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create testimg4@snap 2023-12-24T18:01:46.257 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T18:01:46.267 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd clone --rbd-default-clone-format=2 testimg4@snap testimg6 2023-12-24T18:01:46.347 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap rm testimg4@snap 2023-12-24T18:01:46.400 INFO:tasks.workunit.client.0.smithi087.stderr: Removing snap: 100% complete...done. 2023-12-24T18:01:46.409 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm --rbd_move_parent_to_trash_on_remove=true testimg1 2023-12-24T18:01:46.482 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T18:01:46.485 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm --rbd_move_parent_to_trash_on_remove=true testimg2 2023-12-24T18:01:46.556 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T18:01:46.559 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv testimg3 2023-12-24T18:01:46.615 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm --rbd_move_parent_to_trash_on_remove=true testimg4 2023-12-24T18:01:46.690 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T18:01:46.692 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:46.693 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:46.693 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 4 2023-12-24T18:01:46.720 INFO:tasks.workunit.client.0.smithi087.stdout:4 2023-12-24T18:01:46.720 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash purge 2023-12-24T18:01:46.720 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'some expired images could not be removed' 2023-12-24T18:01:46.841 INFO:tasks.workunit.client.0.smithi087.stdout:rbd: some expired images could not be removed 2023-12-24T18:01:46.842 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:46.842 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:46.842 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 4 2023-12-24T18:01:46.868 INFO:tasks.workunit.client.0.smithi087.stdout:4 2023-12-24T18:01:46.868 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv testimg6 2023-12-24T18:01:46.925 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:46.925 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:46.925 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 5 2023-12-24T18:01:46.952 INFO:tasks.workunit.client.0.smithi087.stdout:5 2023-12-24T18:01:46.953 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash purge 2023-12-24T18:01:46.953 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'some expired images could not be removed' 2023-12-24T18:01:48.197 INFO:tasks.workunit.client.0.smithi087.stdout:rbd: some expired images could not be removed 2023-12-24T18:01:48.198 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:48.198 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:48.198 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 3 2023-12-24T18:01:48.225 INFO:tasks.workunit.client.0.smithi087.stdout:3 2023-12-24T18:01:48.226 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:48.226 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep testimg1 2023-12-24T18:01:48.252 INFO:tasks.workunit.client.0.smithi087.stdout:25e0f571e6da testimg1 2023-12-24T18:01:48.252 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:48.252 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep testimg2 2023-12-24T18:01:48.279 INFO:tasks.workunit.client.0.smithi087.stdout:25e6d9613fa3 testimg2 2023-12-24T18:01:48.279 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:48.280 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep testimg3 2023-12-24T18:01:48.306 INFO:tasks.workunit.client.0.smithi087.stdout:25ec5089e18e testimg3 2023-12-24T18:01:48.306 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash mv testimg5 2023-12-24T18:01:48.366 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:48.367 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:48.367 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 4 2023-12-24T18:01:48.392 INFO:tasks.workunit.client.0.smithi087.stdout:4 2023-12-24T18:01:48.393 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash purge 2023-12-24T18:01:48.393 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'some expired images could not be removed' 2023-12-24T18:01:50.180 INFO:tasks.workunit.client.0.smithi087.stdout:rbd: some expired images could not be removed 2023-12-24T18:01:50.181 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:50.181 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:50.181 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 1 2023-12-24T18:01:50.207 INFO:tasks.workunit.client.0.smithi087.stdout:1 2023-12-24T18:01:50.208 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:50.208 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep testimg3 2023-12-24T18:01:50.235 INFO:tasks.workunit.client.0.smithi087.stdout:25ec5089e18e testimg3 2023-12-24T18:01:50.236 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd trash ls 2023-12-24T18:01:50.236 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{ print $1 }' 2023-12-24T18:01:50.262 INFO:tasks.workunit.client.0.smithi087.stderr:+ ID=25ec5089e18e 2023-12-24T18:01:50.263 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap purge --image-id 25ec5089e18e 2023-12-24T18:01:51.088 INFO:tasks.workunit.client.0.smithi087.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2023-12-24T18:01:51.099 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash purge 2023-12-24T18:01:51.168 INFO:tasks.workunit.client.0.smithi087.stderr: Removing images: 100% complete... Removing images: 100% complete...done. 2023-12-24T18:01:51.171 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls 2023-12-24T18:01:51.171 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:51.171 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 0 2023-12-24T18:01:51.199 INFO:tasks.workunit.client.0.smithi087.stdout:0 2023-12-24T18:01:51.199 INFO:tasks.workunit.client.0.smithi087.stderr:+ test_deep_copy_clone 2023-12-24T18:01:51.200 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'testing deep copy clone...' 2023-12-24T18:01:51.200 INFO:tasks.workunit.client.0.smithi087.stdout:testing deep copy clone... 2023-12-24T18:01:51.200 INFO:tasks.workunit.client.0.smithi087.stderr:+ remove_images 2023-12-24T18:01:51.200 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:51.280 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:51.360 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:51.439 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:51.517 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:51.597 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:51.675 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:51.754 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:51.835 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:51.917 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:51.997 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:52.081 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:52.164 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:52.245 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:52.326 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:52.405 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:52.687 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:52.770 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:01:52.852 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create testimg1 --image-format 2 --size 256 2023-12-24T18:01:52.898 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create testimg1 --snap=snap1 2023-12-24T18:01:53.093 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T18:01:53.102 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap protect testimg1@snap1 2023-12-24T18:01:53.142 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd clone testimg1@snap1 testimg2 2023-12-24T18:01:53.207 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create testimg2@snap2 2023-12-24T18:01:54.100 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T18:01:54.110 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd deep copy testimg2 testimg3 2023-12-24T18:01:54.210 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:01:54.212+0000 7f10afcee700 0 -- 172.21.15.87:0/1015119682 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f10900066e0 msgr2=0x7f1090008b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:01:55.111 INFO:tasks.workunit.client.0.smithi087.stderr: Image deep copy: 1% complete... Image deep copy: 3% complete... Image deep copy: 4% complete... Image deep copy: 6% complete... Image deep copy: 7% complete... Image deep copy: 9% complete... Image deep copy: 10% complete... Image deep copy: 12% complete... Image deep copy: 14% complete... Image deep copy: 15% complete... Image deep copy: 17% complete... Image deep copy: 18% complete... Image deep copy: 20% complete... Image deep copy: 21% complete... Image deep copy: 23% complete... Image deep copy: 25% complete... Image deep copy: 26% complete... Image deep copy: 28% complete... Image deep copy: 29% complete... Image deep copy: 31% complete... Image deep copy: 32% complete... Image deep copy: 34% complete... Image deep copy: 35% complete... Image deep copy: 37% complete... Image deep copy: 39% complete... Image deep copy: 40% complete... Image deep copy: 42% complete... Image deep copy: 43% complete... Image deep copy: 45% complete... Image deep copy: 46% complete... Image deep copy: 48% complete... Image deep copy: 50% complete... Image deep copy: 51% complete... Image deep copy: 53% complete... Image deep copy: 54% complete... Image deep copy: 56% complete... Image deep copy: 57% complete... Image deep copy: 59% complete... Image deep copy: 60% complete... Image deep copy: 62% complete... Image deep copy: 64% complete... Image deep copy: 65% complete... Image deep copy: 67% complete... Image deep copy: 68% complete... Image deep copy: 70% complete... Image deep copy: 71% complete... Image deep copy: 73% complete... Image deep copy: 75% complete... Image deep copy: 76% complete... Image deep copy: 78% complete... Image deep copy: 79% complete... Image deep copy: 81% complete... Image deep copy: 82% complete... Image deep copy: 84% complete... Image deep copy: 85% complete... Image deep copy: 87% complete... Image deep copy: 89% complete... Image deep copy: 90% complete... Image deep copy: 92% complete... Image deep copy: 93% complete... Image deep copy: 95% complete... Image deep copy: 96% complete... Image deep copy: 98% complete... Image deep copy: 100% complete...2023-12-24T18:01:55.114+0000 7f10afcee700 0 -- 172.21.15.87:0/1015119682 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f109403a330 msgr2=0x7f109403c750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:01:55.111 INFO:tasks.workunit.client.0.smithi087.stderr: Image deep copy: 100% complete...done. 2023-12-24T18:01:55.115 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info testimg3 2023-12-24T18:01:55.115 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'size 256 MiB' 2023-12-24T18:01:55.159 INFO:tasks.workunit.client.0.smithi087.stdout: size 256 MiB in 64 objects 2023-12-24T18:01:55.159 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info testimg3 2023-12-24T18:01:55.160 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'parent: rbd/testimg1@snap1' 2023-12-24T18:01:55.201 INFO:tasks.workunit.client.0.smithi087.stdout: parent: rbd/testimg1@snap1 2023-12-24T18:01:55.202 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap ls testimg3 2023-12-24T18:01:55.202 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -v SNAPID 2023-12-24T18:01:55.202 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:55.202 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 1 2023-12-24T18:01:55.241 INFO:tasks.workunit.client.0.smithi087.stdout:1 2023-12-24T18:01:55.242 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap ls testimg3 2023-12-24T18:01:55.242 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '.*snap2.*' 2023-12-24T18:01:55.281 INFO:tasks.workunit.client.0.smithi087.stdout: 76 snap2 256 MiB Sun Dec 24 18:01:55 2023 2023-12-24T18:01:55.282 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info testimg2 2023-12-24T18:01:55.282 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'features:.*deep-flatten' 2023-12-24T18:01:55.322 INFO:tasks.workunit.client.0.smithi087.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten 2023-12-24T18:01:55.323 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info testimg3 2023-12-24T18:01:55.323 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'features:.*deep-flatten' 2023-12-24T18:01:55.362 INFO:tasks.workunit.client.0.smithi087.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten 2023-12-24T18:01:55.363 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd flatten testimg2 2023-12-24T18:01:55.413 INFO:tasks.workunit.client.0.smithi087.stderr: Image flatten: 1% complete... Image flatten: 3% complete... Image flatten: 4% complete... Image flatten: 6% complete... Image flatten: 7% complete... Image flatten: 9% complete... Image flatten: 10% complete... Image flatten: 12% complete... Image flatten: 14% complete... Image flatten: 15% complete... Image flatten: 17% complete... Image flatten: 18% complete... Image flatten: 20% complete... Image flatten: 21% complete... Image flatten: 23% complete... Image flatten: 25% complete... Image flatten: 26% complete... Image flatten: 28% complete... Image flatten: 29% complete... Image flatten: 31% complete... Image flatten: 32% complete... Image flatten: 34% complete... Image flatten: 35% complete... Image flatten: 37% complete... Image flatten: 39% complete... Image flatten: 40% complete... Image flatten: 42% complete... Image flatten: 43% complete... Image flatten: 45% complete... Image flatten: 46% complete... Image flatten: 48% complete... Image flatten: 50% complete... Image flatten: 51% complete... Image flatten: 53% complete... Image flatten: 54% complete... Image flatten: 56% complete... Image flatten: 57% complete... Image flatten: 59% complete... Image flatten: 60% complete... Image flatten: 62% complete... Image flatten: 64% complete... Image flatten: 65% complete... Image flatten: 67% complete... Image flatten: 68% complete... Image flatten: 70% complete... Image flatten: 71% complete... Image flatten: 73% complete... Image flatten: 75% complete... Image flatten: 76% complete... Image flatten: 78% complete... Image flatten: 79% complete... Image flatten: 81% complete... Image flatten: 82% complete... Image flatten: 84% complete... Image flatten: 85% complete... Image flatten: 87% complete... Image flatten: 89% complete... Image flatten: 90% complete... Image flatten: 92% complete... Image flatten: 93% complete... Image flatten: 95% complete... Image flatten: 96% complete... Image flatten: 98% complete... Image flatten: 100% complete...done. 2023-12-24T18:01:55.421 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd flatten testimg3 2023-12-24T18:01:55.471 INFO:tasks.workunit.client.0.smithi087.stderr: Image flatten: 1% complete... Image flatten: 3% complete... Image flatten: 4% complete... Image flatten: 6% complete... Image flatten: 7% complete... Image flatten: 9% complete... Image flatten: 10% complete... Image flatten: 12% complete... Image flatten: 14% complete... Image flatten: 15% complete... Image flatten: 17% complete... Image flatten: 18% complete... Image flatten: 20% complete... Image flatten: 21% complete... Image flatten: 23% complete... Image flatten: 25% complete... Image flatten: 26% complete... Image flatten: 28% complete... Image flatten: 29% complete... Image flatten: 31% complete... Image flatten: 32% complete... Image flatten: 34% complete... Image flatten: 35% complete... Image flatten: 37% complete... Image flatten: 39% complete... Image flatten: 40% complete... Image flatten: 42% complete... Image flatten: 43% complete... Image flatten: 45% complete... Image flatten: 46% complete... Image flatten: 48% complete... Image flatten: 50% complete... Image flatten: 51% complete... Image flatten: 53% complete... Image flatten: 54% complete... Image flatten: 56% complete... Image flatten: 57% complete... Image flatten: 59% complete... Image flatten: 60% complete... Image flatten: 62% complete... Image flatten: 64% complete... Image flatten: 65% complete... Image flatten: 67% complete... Image flatten: 68% complete... Image flatten: 70% complete... Image flatten: 71% complete... Image flatten: 73% complete... Image flatten: 75% complete... Image flatten: 76% complete... Image flatten: 78% complete... Image flatten: 79% complete... Image flatten: 81% complete... Image flatten: 82% complete... Image flatten: 84% complete... Image flatten: 85% complete... Image flatten: 87% complete... Image flatten: 89% complete... Image flatten: 90% complete... Image flatten: 92% complete... Image flatten: 93% complete... Image flatten: 95% complete... Image flatten: 96% complete... Image flatten: 98% complete... Image flatten: 100% complete...done. 2023-12-24T18:01:55.476 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap unprotect testimg1@snap1 2023-12-24T18:01:55.519 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap purge testimg2 2023-12-24T18:01:56.100 INFO:tasks.workunit.client.0.smithi087.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2023-12-24T18:01:56.113 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap purge testimg3 2023-12-24T18:01:57.102 INFO:tasks.workunit.client.0.smithi087.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2023-12-24T18:01:57.115 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm testimg2 2023-12-24T18:01:57.232 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 1% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 12% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 23% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 37% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 48% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 62% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 73% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 87% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 98% complete...2023-12-24T18:01:57.235+0000 7f261bb39700 0 -- 172.21.15.87:0/3757737690 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f25f8006550 msgr2=0x7f25f8011f00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:01:57.232 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T18:01:57.235 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm testimg3 2023-12-24T18:01:57.338 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 1% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 12% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 23% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 37% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 48% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 62% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 73% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 87% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 98% complete...2023-12-24T18:01:57.341+0000 7f93c53bc700 0 -- 172.21.15.87:0/125457349 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f93a0008740 msgr2=0x7f93a000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-24T18:01:57.339 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T18:01:57.341 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap protect testimg1@snap1 2023-12-24T18:01:57.382 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd clone testimg1@snap1 testimg2 2023-12-24T18:01:57.448 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create testimg2@snap2 2023-12-24T18:01:58.106 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T18:01:58.115 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd deep copy --flatten testimg2 testimg3 2023-12-24T18:01:59.240 INFO:tasks.workunit.client.0.smithi087.stderr: Image deep copy: 1% complete... Image deep copy: 3% complete...2023-12-24T18:01:59.242+0000 7ff0f4949700 0 -- 172.21.15.87:0/1621066479 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7ff0d0008740 msgr2=0x7ff0d000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:01:59.324 INFO:tasks.workunit.client.0.smithi087.stderr: Image deep copy: 4% complete... Image deep copy: 6% complete... Image deep copy: 7% complete... Image deep copy: 9% complete...2023-12-24T18:01:59.324+0000 7ff0f514a700 0 -- 172.21.15.87:0/1621066479 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x557e3c2ddda0 msgr2=0x557e3c340780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:01:59.918 INFO:tasks.workunit.client.0.smithi087.stderr: Image deep copy: 10% complete... Image deep copy: 12% complete... Image deep copy: 14% complete... Image deep copy: 15% complete... Image deep copy: 17% complete... Image deep copy: 18% complete... Image deep copy: 20% complete... Image deep copy: 21% complete... Image deep copy: 23% complete... Image deep copy: 25% complete... Image deep copy: 26% complete... Image deep copy: 28% complete... Image deep copy: 29% complete... Image deep copy: 31% complete... Image deep copy: 32% complete... Image deep copy: 34% complete... Image deep copy: 35% complete... Image deep copy: 37% complete... Image deep copy: 39% complete... Image deep copy: 40% complete... Image deep copy: 42% complete... Image deep copy: 43% complete... Image deep copy: 45% complete... Image deep copy: 46% complete... Image deep copy: 48% complete... Image deep copy: 50% complete... Image deep copy: 51% complete... Image deep copy: 53% complete... Image deep copy: 54% complete... Image deep copy: 56% complete... Image deep copy: 57% complete... Image deep copy: 59% complete... Image deep copy: 60% complete... Image deep copy: 62% complete... Image deep copy: 64% complete... Image deep copy: 65% complete... Image deep copy: 67% complete... Image deep copy: 68% complete... Image deep copy: 70% complete... Image deep copy: 71% complete... Image deep copy: 73% complete... Image deep copy: 75% complete... Image deep copy: 76% complete... Image deep copy: 78% complete... Image deep copy: 79% complete... Image deep copy: 81% complete... Image deep copy: 82% complete... Image deep copy: 84% complete... Image deep copy: 85% complete... Image deep copy: 87% complete... Image deep copy: 89% complete... Image deep copy: 90% complete... Image deep copy: 92% complete... Image deep copy: 93% complete... Image deep copy: 95% complete... Image deep copy: 96% complete... Image deep copy: 98% complete... Image deep copy: 100% complete... Image deep copy: 100% complete...done. 2023-12-24T18:01:59.922 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info testimg3 2023-12-24T18:01:59.922 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'size 256 MiB' 2023-12-24T18:01:59.955 INFO:tasks.workunit.client.0.smithi087.stdout: size 256 MiB in 64 objects 2023-12-24T18:01:59.956 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info testimg3 2023-12-24T18:01:59.956 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -v parent: 2023-12-24T18:01:59.989 INFO:tasks.workunit.client.0.smithi087.stdout:rbd image 'testimg3': 2023-12-24T18:01:59.990 INFO:tasks.workunit.client.0.smithi087.stdout: size 256 MiB in 64 objects 2023-12-24T18:01:59.990 INFO:tasks.workunit.client.0.smithi087.stdout: order 22 (4 MiB objects) 2023-12-24T18:01:59.990 INFO:tasks.workunit.client.0.smithi087.stdout: snapshot_count: 1 2023-12-24T18:01:59.990 INFO:tasks.workunit.client.0.smithi087.stdout: id: 26f9e0468bda 2023-12-24T18:01:59.990 INFO:tasks.workunit.client.0.smithi087.stdout: block_name_prefix: rbd_data.26f9e0468bda 2023-12-24T18:01:59.990 INFO:tasks.workunit.client.0.smithi087.stdout: format: 2 2023-12-24T18:01:59.990 INFO:tasks.workunit.client.0.smithi087.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten 2023-12-24T18:01:59.990 INFO:tasks.workunit.client.0.smithi087.stdout: op_features: 2023-12-24T18:01:59.990 INFO:tasks.workunit.client.0.smithi087.stdout: flags: 2023-12-24T18:01:59.990 INFO:tasks.workunit.client.0.smithi087.stdout: create_timestamp: Sun Dec 24 18:01:58 2023 2023-12-24T18:01:59.990 INFO:tasks.workunit.client.0.smithi087.stdout: access_timestamp: Sun Dec 24 18:01:58 2023 2023-12-24T18:01:59.990 INFO:tasks.workunit.client.0.smithi087.stdout: modify_timestamp: Sun Dec 24 18:01:58 2023 2023-12-24T18:01:59.991 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap ls testimg3 2023-12-24T18:01:59.991 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -v SNAPID 2023-12-24T18:01:59.991 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:01:59.991 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 1 2023-12-24T18:02:00.024 INFO:tasks.workunit.client.0.smithi087.stdout:1 2023-12-24T18:02:00.024 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap ls testimg3 2023-12-24T18:02:00.024 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '.*snap2.*' 2023-12-24T18:02:00.058 INFO:tasks.workunit.client.0.smithi087.stdout: 80 snap2 256 MiB Sun Dec 24 18:01:59 2023 2023-12-24T18:02:00.059 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd info testimg2 2023-12-24T18:02:00.059 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'features:.*deep-flatten' 2023-12-24T18:02:00.098 INFO:tasks.workunit.client.0.smithi087.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten 2023-12-24T18:02:00.098 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd flatten testimg2 2023-12-24T18:02:00.148 INFO:tasks.workunit.client.0.smithi087.stderr: Image flatten: 1% complete... Image flatten: 3% complete... Image flatten: 4% complete... Image flatten: 6% complete... Image flatten: 7% complete... Image flatten: 9% complete... Image flatten: 10% complete... Image flatten: 12% complete... Image flatten: 14% complete... Image flatten: 15% complete... Image flatten: 17% complete... Image flatten: 18% complete... Image flatten: 20% complete... Image flatten: 21% complete... Image flatten: 23% complete... Image flatten: 25% complete... Image flatten: 26% complete... Image flatten: 28% complete... Image flatten: 29% complete... Image flatten: 31% complete... Image flatten: 32% complete... Image flatten: 34% complete... Image flatten: 35% complete... Image flatten: 37% complete... Image flatten: 39% complete... Image flatten: 40% complete... Image flatten: 42% complete... Image flatten: 43% complete... Image flatten: 45% complete... Image flatten: 46% complete... Image flatten: 48% complete... Image flatten: 50% complete... Image flatten: 51% complete... Image flatten: 53% complete... Image flatten: 54% complete... Image flatten: 56% complete... Image flatten: 57% complete... Image flatten: 59% complete... Image flatten: 60% complete... Image flatten: 62% complete... Image flatten: 64% complete... Image flatten: 65% complete... Image flatten: 67% complete... Image flatten: 68% complete... Image flatten: 70% complete... Image flatten: 71% complete... Image flatten: 73% complete... Image flatten: 75% complete... Image flatten: 76% complete... Image flatten: 78% complete... Image flatten: 79% complete... Image flatten: 81% complete... Image flatten: 82% complete... Image flatten: 84% complete... Image flatten: 85% complete... Image flatten: 87% complete... Image flatten: 89% complete... Image flatten: 90% complete... Image flatten: 92% complete... Image flatten: 93% complete... Image flatten: 95% complete... Image flatten: 96% complete... Image flatten: 98% complete... Image flatten: 100% complete...done. 2023-12-24T18:02:00.156 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap unprotect testimg1@snap1 2023-12-24T18:02:00.198 INFO:tasks.workunit.client.0.smithi087.stderr:+ remove_images 2023-12-24T18:02:00.198 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:01.246 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:02.214 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:03.321 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:03.405 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:03.686 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:03.764 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:03.844 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:03.927 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:04.006 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:04.092 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:04.174 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:04.253 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:04.334 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:04.416 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:04.496 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:04.579 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:04.861 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:04.944 INFO:tasks.workunit.client.0.smithi087.stderr:+ test_clone_v2 2023-12-24T18:02:04.944 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'testing clone v2...' 2023-12-24T18:02:04.944 INFO:tasks.workunit.client.0.smithi087.stdout:testing clone v2... 2023-12-24T18:02:04.945 INFO:tasks.workunit.client.0.smithi087.stderr:+ remove_images 2023-12-24T18:02:04.945 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:05.026 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:05.106 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:05.184 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:05.264 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:05.344 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:05.423 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:05.505 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:05.587 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:05.668 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:05.748 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:05.827 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:05.907 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:05.986 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:06.067 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:06.148 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:06.228 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:06.310 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:06.391 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 -s 1 test1 2023-12-24T18:02:06.441 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create test1@1 2023-12-24T18:02:07.127 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T18:02:07.136 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd clone --rbd-default-clone-format=1 test1@1 test2 2023-12-24T18:02:07.175 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:02:07.178+0000 7f73f7fff700 -1 librbd::image::CloneRequest: 0x55dc56b9b7e0 validate_parent: parent snapshot must be protected 2023-12-24T18:02:07.176 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: clone error: (22) Invalid argument 2023-12-24T18:02:07.178 INFO:tasks.workunit.client.0.smithi087.stderr:+ true 2023-12-24T18:02:07.178 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd clone --rbd-default-clone-format=2 test1@1 test2 2023-12-24T18:02:07.245 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd clone --rbd-default-clone-format=2 test1@1 test3 2023-12-24T18:02:07.308 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap protect test1@1 2023-12-24T18:02:07.349 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd clone --rbd-default-clone-format=1 test1@1 test4 2023-12-24T18:02:07.421 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd children test1@1 2023-12-24T18:02:07.421 INFO:tasks.workunit.client.0.smithi087.stderr:+ sort 2023-12-24T18:02:07.422 INFO:tasks.workunit.client.0.smithi087.stderr:+ tr '\n' ' ' 2023-12-24T18:02:07.422 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -E 'test2.*test3.*test4' 2023-12-24T18:02:07.461 INFO:tasks.workunit.client.0.smithi087.stdout:rbd/test2 rbd/test3 rbd/test4 2023-12-24T18:02:07.462 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd children --descendants test1 2023-12-24T18:02:07.462 INFO:tasks.workunit.client.0.smithi087.stderr:+ sort 2023-12-24T18:02:07.462 INFO:tasks.workunit.client.0.smithi087.stderr:+ tr '\n' ' ' 2023-12-24T18:02:07.462 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -E 'test2.*test3.*test4' 2023-12-24T18:02:07.513 INFO:tasks.workunit.client.0.smithi087.stdout:rbd/test2 rbd/test3 rbd/test4 2023-12-24T18:02:07.514 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd remove test4 2023-12-24T18:02:07.610 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:02:07.613+0000 7f0148dc3700 0 -- 172.21.15.87:0/2086467566 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f0124008740 msgr2=0x7f012400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:02:07.617 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T18:02:07.618 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap unprotect test1@1 2023-12-24T18:02:07.661 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap remove test1@1 2023-12-24T18:02:07.702 INFO:tasks.workunit.client.0.smithi087.stderr: Removing snap: 100% complete...done. 2023-12-24T18:02:07.710 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap list --all test1 2023-12-24T18:02:07.710 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep -E 'trash \(1\) *$' 2023-12-24T18:02:07.746 INFO:tasks.workunit.client.0.smithi087.stdout: 84 97d64005-c1d8-44d2-99da-458fdaf789de 1 MiB Sun Dec 24 18:02:07 2023 trash (1) 2023-12-24T18:02:07.746 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create test1@2 2023-12-24T18:02:08.126 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T18:02:08.134 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test1 2023-12-24T18:02:08.134 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'image has snapshots' 2023-12-24T18:02:08.189 INFO:tasks.workunit.client.0.smithi087.stdout:rbd: image has snapshots - these must be deleted with 'rbd snap purge' before the image can be removed. 2023-12-24T18:02:08.189 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap rm test1@2 2023-12-24T18:02:09.133 INFO:tasks.workunit.client.0.smithi087.stderr: Removing snap: 100% complete...done. 2023-12-24T18:02:09.143 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test1 2023-12-24T18:02:09.143 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'linked clones' 2023-12-24T18:02:09.202 INFO:tasks.workunit.client.0.smithi087.stdout:rbd: image has snapshots with linked clones - these must be deleted or flattened before the image can be removed. 2023-12-24T18:02:09.202 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test3 2023-12-24T18:02:09.300 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:02:09.303+0000 7f075856f700 0 -- 172.21.15.87:0/2610130987 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f0734008740 msgr2=0x7f073400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:02:09.312 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T18:02:09.315 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test1 2023-12-24T18:02:09.315 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep 'linked clones' 2023-12-24T18:02:09.369 INFO:tasks.workunit.client.0.smithi087.stdout:rbd: image has snapshots with linked clones - these must be deleted or flattened before the image can be removed. 2023-12-24T18:02:09.370 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd flatten test2 2023-12-24T18:02:10.143 INFO:tasks.workunit.client.0.smithi087.stderr: Image flatten: 100% complete...done. 2023-12-24T18:02:10.154 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap list --all test1 2023-12-24T18:02:10.155 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:02:10.155 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^0$' 2023-12-24T18:02:10.192 INFO:tasks.workunit.client.0.smithi087.stdout:0 2023-12-24T18:02:10.193 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test1 2023-12-24T18:02:10.298 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:02:10.301+0000 7fa14e658700 0 -- 172.21.15.87:0/979778398 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fa134008740 msgr2=0x7fa13400abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:02:10.299 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T18:02:10.301 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test2 2023-12-24T18:02:10.408 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:02:10.411+0000 7f03a21f3700 0 -- 172.21.15.87:0/844954609 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f0388006530 msgr2=0x7f03880148c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:02:10.409 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T18:02:10.411 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 -s 1 test1 2023-12-24T18:02:10.455 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create test1@1 2023-12-24T18:02:11.141 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T18:02:11.149 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create test1@2 2023-12-24T18:02:12.095 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T18:02:12.106 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd clone test1@1 test2 --rbd-default-clone-format 2 2023-12-24T18:02:12.175 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd clone test1@2 test3 --rbd-default-clone-format 2 2023-12-24T18:02:12.254 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap rm test1@1 2023-12-24T18:02:12.295 INFO:tasks.workunit.client.0.smithi087.stderr: Removing snap: 100% complete...done. 2023-12-24T18:02:12.302 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap rm test1@2 2023-12-24T18:02:12.345 INFO:tasks.workunit.client.0.smithi087.stderr: Removing snap: 100% complete...done. 2023-12-24T18:02:12.351 INFO:tasks.workunit.client.0.smithi087.stderr:+ expect_fail rbd rm test1 2023-12-24T18:02:12.352 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test1 2023-12-24T18:02:12.397 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:02:12.400+0000 7f4454dc8540 -1 librbd::api::Image: remove: image has snapshots - not removing 2023-12-24T18:02:12.397 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 0% complete...failed. 2023-12-24T18:02:12.403 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: image has snapshots with linked clones - these must be deleted or flattened before the image can be removed. 2023-12-24T18:02:12.406 INFO:tasks.workunit.client.0.smithi087.stderr:+ return 0 2023-12-24T18:02:12.406 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test1 --rbd-move-parent-to-trash-on-remove=true 2023-12-24T18:02:12.481 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T18:02:12.483 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls -a 2023-12-24T18:02:12.483 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep test1 2023-12-24T18:02:12.511 INFO:tasks.workunit.client.0.smithi087.stdout:282d3f47c8e8 test1 2023-12-24T18:02:12.511 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test2 2023-12-24T18:02:12.606 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:02:12.610+0000 7f7f9be30700 0 -- 172.21.15.87:0/4149792839 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f7f80008740 msgr2=0x7f7f8000abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:02:12.627 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:02:12.631+0000 7f7f9c631700 0 -- 172.21.15.87:0/4149792839 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x5559e80e8b50 msgr2=0x5559e818eeb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:02:13.173 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T18:02:13.176 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls -a 2023-12-24T18:02:13.176 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep test1 2023-12-24T18:02:13.204 INFO:tasks.workunit.client.0.smithi087.stdout:282d3f47c8e8 test1 2023-12-24T18:02:13.204 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test3 2023-12-24T18:02:13.300 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:02:13.303+0000 7fd30b9e1700 0 -- 172.21.15.87:0/2398321234 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x55563bc800a0 msgr2=0x55563bca6c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:02:13.312 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:02:13.316+0000 7fd30c1e2700 0 -- 172.21.15.87:0/2398321234 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x55563bb8eb50 msgr2=0x55563bc35910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:02:14.190 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T18:02:14.192 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd trash ls -a 2023-12-24T18:02:14.192 INFO:tasks.workunit.client.0.smithi087.stderr:+ expect_fail grep test1 2023-12-24T18:02:14.192 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep test1 2023-12-24T18:02:14.221 INFO:tasks.workunit.client.0.smithi087.stderr:+ return 0 2023-12-24T18:02:14.222 INFO:tasks.workunit.client.0.smithi087.stderr:+ test_thick_provision 2023-12-24T18:02:14.222 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'testing thick provision...' 2023-12-24T18:02:14.222 INFO:tasks.workunit.client.0.smithi087.stdout:testing thick provision... 2023-12-24T18:02:14.222 INFO:tasks.workunit.client.0.smithi087.stderr:+ remove_images 2023-12-24T18:02:14.222 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:14.304 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:14.385 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:14.465 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:14.545 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:14.626 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:14.909 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:14.989 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:15.070 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:15.152 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:15.237 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:15.317 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:15.398 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:15.478 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:15.559 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:15.641 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:15.721 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:15.803 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:15.887 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 --thick-provision -s 64M test1 2023-12-24T18:02:16.228 INFO:tasks.workunit.client.0.smithi087.stderr: Thick provisioning: 6% complete... Thick provisioning: 12% complete... Thick provisioning: 18% complete... Thick provisioning: 25% complete... Thick provisioning: 31% complete... Thick provisioning: 37% complete... Thick provisioning: 43% complete... Thick provisioning: 50% complete... Thick provisioning: 56% complete... Thick provisioning: 62% complete... Thick provisioning: 68% complete... Thick provisioning: 75% complete... Thick provisioning: 81% complete... Thick provisioning: 87% complete... Thick provisioning: 93% complete... Thick provisioning: 100% complete... Thick provisioning: 100% complete...done. 2023-12-24T18:02:16.238 INFO:tasks.workunit.client.0.smithi087.stderr:+ count=0 2023-12-24T18:02:16.240 INFO:tasks.workunit.client.0.smithi087.stderr:+ ret= 2023-12-24T18:02:16.241 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 0 -lt 10 ']' 2023-12-24T18:02:16.241 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd du 2023-12-24T18:02:16.241 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep test1 2023-12-24T18:02:16.241 INFO:tasks.workunit.client.0.smithi087.stderr:+ tr -s ' ' 2023-12-24T18:02:16.241 INFO:tasks.workunit.client.0.smithi087.stderr:+ cut -d ' ' -f 4-5 2023-12-24T18:02:16.241 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^64 MiB' 2023-12-24T18:02:16.271 INFO:tasks.workunit.client.0.smithi087.stdout:64 MiB 2023-12-24T18:02:16.272 INFO:tasks.workunit.client.0.smithi087.stderr:+ ret=0 2023-12-24T18:02:16.272 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 0 = 0 ']' 2023-12-24T18:02:16.272 INFO:tasks.workunit.client.0.smithi087.stderr:+ break 2023-12-24T18:02:16.272 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd du 2023-12-24T18:02:16.301 INFO:tasks.workunit.client.0.smithi087.stdout:NAME PROVISIONED USED 2023-12-24T18:02:16.302 INFO:tasks.workunit.client.0.smithi087.stdout:test1 64 MiB 64 MiB 2023-12-24T18:02:16.304 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 0 '!=' 0 ']' 2023-12-24T18:02:16.304 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test1 2023-12-24T18:02:16.415 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 6% complete... Removing image: 12% complete... Removing image: 18% complete... Removing image: 25% complete... Removing image: 31% complete... Removing image: 37% complete... Removing image: 43% complete... Removing image: 50% complete... Removing image: 56% complete... Removing image: 62% complete... Removing image: 68% complete... Removing image: 75% complete... Removing image: 81% complete... Removing image: 87% complete... Removing image: 93% complete...2023-12-24T18:02:16.419+0000 7f7c07ec0700 0 -- 172.21.15.87:0/2782750706 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x55612ac35980 msgr2=0x55612acdbe20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:02:16.448 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:02:16.451+0000 7f7c076bf700 0 -- 172.21.15.87:0/2782750706 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f7be8006750 msgr2=0x7f7be8008bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:02:16.448 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T18:02:16.451 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T18:02:16.451 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep test1 2023-12-24T18:02:16.451 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:02:16.451 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^0$' 2023-12-24T18:02:16.477 INFO:tasks.workunit.client.0.smithi087.stdout:0 2023-12-24T18:02:16.477 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 --thick-provision -s 4G test1 2023-12-24T18:02:16.912 INFO:tasks.workunit.client.0.smithi087.stderr: Thick provisioning: 1% complete... Thick provisioning: 2% complete... Thick provisioning: 3% complete... Thick provisioning: 4% complete... Thick provisioning: 5% complete...2023-12-24T18:02:16.915+0000 7fd573e1e700 0 -- 172.21.15.87:0/2439164553 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x5587355dff40 msgr2=0x558735684550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:02:17.033 INFO:tasks.workunit.client.0.smithi087.stderr: Thick provisioning: 6% complete...2023-12-24T18:02:17.037+0000 7fd573e1e700 0 -- 172.21.15.87:0/2439164553 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fd55803a520 msgr2=0x7fd55803c940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:02:21.878 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:02:21.882+0000 7f7f67365700 -1 reset not still connected to 0x55ee3edc1ee0 2023-12-24T18:02:26.774 INFO:tasks.workunit.client.0.smithi087.stderr: Thick provisioning: 7% complete... Thick provisioning: 8% complete... Thick provisioning: 9% complete... Thick provisioning: 10% complete... Thick provisioning: 11% complete... Thick provisioning: 12% complete... Thick provisioning: 13% complete... Thick provisioning: 14% complete... Thick provisioning: 15% complete... Thick provisioning: 16% complete... Thick provisioning: 17% complete... Thick provisioning: 18% complete... Thick provisioning: 19% complete... Thick provisioning: 20% complete... Thick provisioning: 21% complete... Thick provisioning: 22% complete... Thick provisioning: 23% complete... Thick provisioning: 24% complete... Thick provisioning: 25% complete... Thick provisioning: 26% complete... Thick provisioning: 27% complete... Thick provisioning: 28% complete... Thick provisioning: 29% complete... Thick provisioning: 30% complete... Thick provisioning: 31% complete... Thick provisioning: 32% complete... Thick provisioning: 33% complete... Thick provisioning: 34% complete... Thick provisioning: 35% complete... Thick provisioning: 36% complete... Thick provisioning: 37% complete... Thick provisioning: 38% complete... Thick provisioning: 39% complete... Thick provisioning: 40% complete... Thick provisioning: 41% complete... Thick provisioning: 42% complete... Thick provisioning: 43% complete... Thick provisioning: 44% complete... Thick provisioning: 45% complete... Thick provisioning: 46% complete... Thick provisioning: 47% complete... Thick provisioning: 48% complete... Thick provisioning: 49% complete... Thick provisioning: 50% complete... Thick provisioning: 51% complete... Thick provisioning: 52% complete... Thick provisioning: 53% complete... Thick provisioning: 54% complete... Thick provisioning: 55% complete... Thick provisioning: 56% complete... Thick provisioning: 57% complete... Thick provisioning: 58% complete... Thick provisioning: 59% complete... Thick provisioning: 60% complete... Thick provisioning: 61% complete... Thick provisioning: 62% complete... Thick provisioning: 63% complete... Thick provisioning: 64% complete... Thick provisioning: 65% complete... Thick provisioning: 66% complete... Thick provisioning: 67% complete... Thick provisioning: 68% complete... Thick provisioning: 69% complete... Thick provisioning: 70% complete... Thick provisioning: 71% complete... Thick provisioning: 72% complete... Thick provisioning: 73% complete... Thick provisioning: 74% complete... Thick provisioning: 75% complete... Thick provisioning: 76% complete... Thick provisioning: 77% complete... Thick provisioning: 78% complete... Thick provisioning: 79% complete... Thick provisioning: 80% complete... Thick provisioning: 81% complete... Thick provisioning: 82% complete... Thick provisioning: 83% complete... Thick provisioning: 84% complete... Thick provisioning: 85% complete... Thick provisioning: 86% complete... Thick provisioning: 87% complete... Thick provisioning: 88% complete... Thick provisioning: 89% complete... Thick provisioning: 90% complete... Thick provisioning: 91% complete... Thick provisioning: 92% complete... Thick provisioning: 93% complete... Thick provisioning: 94% complete... Thick provisioning: 95% complete... Thick provisioning: 96% complete... Thick provisioning: 97% complete... Thick provisioning: 98% complete... Thick provisioning: 99% complete... Thick provisioning: 100% complete... Thick provisioning: 100% complete...done. 2023-12-24T18:02:26.785 INFO:tasks.workunit.client.0.smithi087.stderr:+ count=0 2023-12-24T18:02:26.785 INFO:tasks.workunit.client.0.smithi087.stderr:+ ret= 2023-12-24T18:02:26.785 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 0 -lt 10 ']' 2023-12-24T18:02:26.785 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd du 2023-12-24T18:02:26.785 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep test1 2023-12-24T18:02:26.785 INFO:tasks.workunit.client.0.smithi087.stderr:+ tr -s ' ' 2023-12-24T18:02:26.785 INFO:tasks.workunit.client.0.smithi087.stderr:+ cut -d ' ' -f 4-5 2023-12-24T18:02:26.785 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^4 GiB' 2023-12-24T18:02:26.818 INFO:tasks.workunit.client.0.smithi087.stdout:4 GiB 2023-12-24T18:02:26.819 INFO:tasks.workunit.client.0.smithi087.stderr:+ ret=0 2023-12-24T18:02:26.819 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 0 = 0 ']' 2023-12-24T18:02:26.819 INFO:tasks.workunit.client.0.smithi087.stderr:+ break 2023-12-24T18:02:26.819 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd du 2023-12-24T18:02:26.849 INFO:tasks.workunit.client.0.smithi087.stdout:NAME PROVISIONED USED 2023-12-24T18:02:26.849 INFO:tasks.workunit.client.0.smithi087.stdout:test1 4 GiB 4 GiB 2023-12-24T18:02:26.851 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 0 '!=' 0 ']' 2023-12-24T18:02:26.851 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm test1 2023-12-24T18:02:26.957 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 1% complete... Removing image: 2% complete...2023-12-24T18:02:26.960+0000 7fd0fc383700 0 -- 172.21.15.87:0/1971177994 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fd0dc005fd0 msgr2=0x7fd0dc008460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:02:26.974 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 3% complete...2023-12-24T18:02:26.977+0000 7fd0fcb84700 0 -- 172.21.15.87:0/1971177994 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x55ad122cc980 msgr2=0x55ad12372c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:02:27.877 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-24T18:02:27.881 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep test1 2023-12-24T18:02:27.881 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:02:27.881 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd ls 2023-12-24T18:02:27.883 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^0$' 2023-12-24T18:02:27.909 INFO:tasks.workunit.client.0.smithi087.stdout:0 2023-12-24T18:02:27.909 INFO:tasks.workunit.client.0.smithi087.stderr:+ test_namespace 2023-12-24T18:02:27.909 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'testing namespace...' 2023-12-24T18:02:27.909 INFO:tasks.workunit.client.0.smithi087.stdout:testing namespace... 2023-12-24T18:02:27.910 INFO:tasks.workunit.client.0.smithi087.stderr:+ remove_images 2023-12-24T18:02:27.910 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:27.988 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:28.069 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:28.151 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:28.232 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:28.313 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:28.393 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:28.475 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:28.554 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:28.635 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:28.718 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:28.799 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:28.881 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:29.162 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:29.245 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:29.329 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:29.410 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:29.492 INFO:tasks.workunit.client.0.smithi087.stderr:+ for img in $IMGS 2023-12-24T18:02:29.574 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd namespace ls 2023-12-24T18:02:29.574 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:02:29.574 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^0$' 2023-12-24T18:02:29.601 INFO:tasks.workunit.client.0.smithi087.stdout:0 2023-12-24T18:02:29.601 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd namespace create rbd/test1 2023-12-24T18:02:29.634 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd namespace create --pool rbd --namespace test2 2023-12-24T18:02:29.668 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd namespace create --namespace test3 2023-12-24T18:02:29.702 INFO:tasks.workunit.client.0.smithi087.stderr:+ expect_fail rbd namespace create rbd/test3 2023-12-24T18:02:29.702 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd namespace create rbd/test3 2023-12-24T18:02:29.731 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: failed to created namespace: (17) File exists 2023-12-24T18:02:29.731 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:02:29.734+0000 7f15b6f6f540 -1 librbd::api::Namespace: create: failed to add namespace: (17) File exists 2023-12-24T18:02:29.734 INFO:tasks.workunit.client.0.smithi087.stderr:+ return 0 2023-12-24T18:02:29.734 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd namespace list 2023-12-24T18:02:29.734 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep test 2023-12-24T18:02:29.734 INFO:tasks.workunit.client.0.smithi087.stderr:+ wc -l 2023-12-24T18:02:29.734 INFO:tasks.workunit.client.0.smithi087.stderr:+ grep '^3$' 2023-12-24T18:02:29.760 INFO:tasks.workunit.client.0.smithi087.stdout:3 2023-12-24T18:02:29.761 INFO:tasks.workunit.client.0.smithi087.stderr:+ expect_fail rbd namespace remove --pool rbd missing 2023-12-24T18:02:29.761 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd namespace remove --pool rbd missing 2023-12-24T18:02:29.783 INFO:tasks.workunit.client.0.smithi087.stderr:rbd: namespace name was not specified 2023-12-24T18:02:29.784 INFO:tasks.workunit.client.0.smithi087.stderr:+ return 0 2023-12-24T18:02:29.784 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 --size 1G rbd/test1/image1 2023-12-24T18:02:29.829 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd bench --io-type write --io-pattern rand --io-total 32M --io-size 4K rbd/test1/image1 2023-12-24T18:02:29.882 INFO:tasks.workunit.client.0.smithi087.stdout:bench type write io_size 4096 io_threads 16 bytes 33554432 pattern random 2023-12-24T18:02:29.914 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:02:29.915+0000 7f762e76c700 0 -- 172.21.15.87:0/1402865748 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x55e08e87caa0 msgr2=0x55e08e891bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-24T18:02:30.681 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:02:30.684+0000 7f762df6b700 0 -- 172.21.15.87:0/1402865748 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x55e08e8df440 msgr2=0x7f761003ceb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:02:31.055 INFO:tasks.workunit.client.0.smithi087.stdout: SEC OPS OPS/SEC BYTES/SEC 2023-12-24T18:02:31.055 INFO:tasks.workunit.client.0.smithi087.stdout: 1 3600 3085.39 12 MiB/s 2023-12-24T18:02:31.088 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:02:31.090+0000 7f762df6b700 0 -- 172.21.15.87:0/1402865748 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x55e08e8df440 msgr2=0x7f761003ceb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:02:31.395 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:02:31.397+0000 7f762df6b700 0 -- 172.21.15.87:0/1402865748 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7f761003a460 msgr2=0x7f761003c840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-24T18:02:31.680 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:02:31.683+0000 7f762e76c700 0 -- 172.21.15.87:0/1402865748 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f761003a460 msgr2=0x7f761003c840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-24T18:02:31.760 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:02:31.762+0000 7f762df6b700 0 -- 172.21.15.87:0/1402865748 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x55e08e8df440 msgr2=0x7f761003ceb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:02:32.103 INFO:tasks.workunit.client.0.smithi087.stdout: 2 4160 1881.12 7.3 MiB/s 2023-12-24T18:02:32.985 INFO:tasks.workunit.client.0.smithi087.stdout: 3 5184 1676.38 6.5 MiB/s 2023-12-24T18:02:33.318 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:02:33.320+0000 7f762e76c700 0 -- 172.21.15.87:0/1402865748 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x55e08e87caa0 msgr2=0x7f76100417c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:02:33.985 INFO:tasks.workunit.client.0.smithi087.stdout: 4 6528 1594.97 6.2 MiB/s 2023-12-24T18:02:34.099 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:02:34.101+0000 7f762df6b700 0 -- 172.21.15.87:0/1402865748 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x55e08e87caa0 msgr2=0x7f761003c840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:02:34.981 INFO:tasks.workunit.client.0.smithi087.stdout: 5 8144 1600.35 6.3 MiB/s 2023-12-24T18:02:35.494 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:02:35.496+0000 7f762e76c700 0 -- 172.21.15.87:0/1402865748 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f761003a460 msgr2=0x7f76100484d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:02:35.567 INFO:tasks.workunit.client.0.smithi087.stdout:elapsed: 5 ops: 8192 ops/sec: 1440.76 bytes/sec: 5.6 MiB/s 2023-12-24T18:02:35.577 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap create rbd/test1/image1@1 2023-12-24T18:02:36.125 INFO:tasks.workunit.client.0.smithi087.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-12-24T18:02:36.134 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd clone --rbd-default-clone-format 2 rbd/test1/image1@1 rbd/test2/image1 2023-12-24T18:02:36.215 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd snap rm rbd/test1/image1@1 2023-12-24T18:02:36.257 INFO:tasks.workunit.client.0.smithi087.stderr: Removing snap: 100% complete...done. 2023-12-24T18:02:36.265 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd export rbd/test1/image1 - 2023-12-24T18:02:36.265 INFO:tasks.workunit.client.0.smithi087.stderr:+ cmp /dev/fd/63 /dev/fd/62 2023-12-24T18:02:36.265 INFO:tasks.workunit.client.0.smithi087.stderr:++ rbd export rbd/test2/image1 - 2023-12-24T18:02:36.462 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete...2023-12-24T18:02:36.465+0000 7f1584dfc700 0 -- 172.21.15.87:0/224319732 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x55d3cc7961a0 msgr2=0x55d3cc798580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:02:36.515 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 16% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete...2023-12-24T18:02:36.517+0000 7f15855fd700 0 -- 172.21.15.87:0/224319732 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x55d3cc6a03b0 msgr2=0x55d3cc748b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:02:36.648 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete...2023-12-24T18:02:36.650+0000 7f3863f66700 0 -- 172.21.15.87:0/708124170 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x55a663f43f70 msgr2=0x55a663f46350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:02:36.741 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete...2023-12-24T18:02:36.744+0000 7f3863f66700 0 -- 172.21.15.87:0/708124170 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7f384803a270 msgr2=0x7f384803c650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:02:37.475 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-24T18:02:37.476 INFO:tasks.workunit.client.0.smithi087.stderr: Exporting image: 100% complete...done. 2023-12-24T18:02:37.481 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd rm rbd/test2/image1 2023-12-24T18:02:37.580 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete...2023-12-24T18:02:37.583+0000 7f866ce42700 0 -- 172.21.15.87:0/1490880045 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f8648006520 msgr2=0x7f86480087b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:02:37.594 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:02:37.596+0000 7f866ce42700 0 -- 172.21.15.87:0/1490880045 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7f865003a340 msgr2=0x7f865003c720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:02:38.155 INFO:tasks.workunit.client.0.smithi087.stderr: Removing image: 100% complete...done. 2023-12-24T18:02:38.157 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd create --image-format 2 --size 1G rbd/image2 2023-12-24T18:02:38.202 INFO:tasks.workunit.client.0.smithi087.stderr:+ rbd bench --io-type write --io-pattern rand --io-total 32M --io-size 4K rbd/image2 2023-12-24T18:02:38.269 INFO:tasks.workunit.client.0.smithi087.stdout:bench type write io_size 4096 io_threads 16 bytes 33554432 pattern random 2023-12-24T18:02:38.343 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:02:38.345+0000 7fccf4bd9700 0 -- 172.21.15.87:0/3193322219 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fccd4007900 msgr2=0x7fccd4009ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-24T18:02:39.389 INFO:tasks.workunit.client.0.smithi087.stdout: SEC OPS OPS/SEC BYTES/SEC 2023-12-24T18:02:39.389 INFO:tasks.workunit.client.0.smithi087.stdout: 1 3296 2957.21 12 MiB/s 2023-12-24T18:02:40.376 INFO:tasks.workunit.client.0.smithi087.stdout: 2 3392 1618.27 6.3 MiB/s 2023-12-24T18:02:40.684 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:02:40.687+0000 7fccf53da700 0 -- 172.21.15.87:0/3193322219 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x564162b40a50 msgr2=0x7fccd803d4d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:02:40.981 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:02:40.982+0000 7fccf4bd9700 0 -- 172.21.15.87:0/3193322219 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fccd4007900 msgr2=0x7fccd803cf90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:02:41.305 INFO:tasks.workunit.client.0.smithi087.stdout: 3 4000 1322.82 5.2 MiB/s 2023-12-24T18:02:42.273 INFO:tasks.workunit.client.0.smithi087.stdout: 4 5040 1263.4 4.9 MiB/s 2023-12-24T18:03:12.638 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:12.641+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 161 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:12.648 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:12.651+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.10614.0:4289 3.2 3:553e7168:::rbd_data.2973596cc8be.00000000000000b5:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1712128~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:13.604 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:13.606+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:13.641 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:13.644+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:14.021 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:14.023+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:14.583 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:14.585+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:14.675 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:14.678+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:15.047 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:15.050+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:15.580 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:15.582+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:15.700 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:15.702+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:16.073 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:16.075+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:16.570 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:16.572+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:16.692 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:16.694+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:17.088 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:17.091+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:17.614 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:17.617+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:17.691 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:17.694+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:18.100 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:18.102+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:18.664 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:18.667+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:18.715 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:18.718+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:19.068 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:19.070+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:19.693 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:19.696+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:19.741 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:19.743+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:20.040 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:20.043+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:20.685 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:20.687+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:20.743 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:20.746+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:20.999 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:21.002+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:21.717 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:21.720+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:21.756 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:21.758+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:22.004 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:22.007+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:22.752 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:22.755+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:22.768 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:22.771+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:22.969 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:22.972+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:23.717 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:23.719+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:23.780 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:23.783+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:23.940 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:23.943+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:24.706 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:24.708+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:24.769 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:24.772+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:24.938 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:24.941+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:25.743 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:25.746+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:25.812 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:25.814+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:25.896 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:25.899+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:26.787 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:26.790+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:26.827 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:26.830+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:26.888 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:26.890+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:27.739 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:27.742+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:27.857 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:27.860+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:27.869 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:27.871+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:28.755 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:28.758+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:28.856 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:28.859+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:28.860 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:28.863+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:29.773 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:29.776+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:29.843 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:29.845+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:29.851 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:29.854+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:30.756 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:30.758+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:30.812 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:30.815+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:30.837 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:30.840+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:31.799 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:31.802+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:31.811 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:31.814+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:31.883 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:31.886+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:32.784 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:32.786+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:32.844 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:32.847+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:32.850 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:32.853+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:33.796 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:33.799+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:33.829 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:33.831+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:33.838 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:33.840+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:34.751 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:34.753+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:34.784 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:34.786+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:34.844 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:34.847+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:35.716 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:35.719+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:35.772 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:35.775+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:35.875 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:35.878+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:36.718 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:36.721+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:36.726 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:36.729+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:36.883 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:36.886+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:37.696 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:37.699+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:37.748 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:37.751+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:37.850 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:37.853+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:38.647 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:38.650+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:38.774 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:38.777+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:38.852 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:38.855+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:39.627 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:39.630+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:39.750 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:39.753+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:39.816 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:39.819+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:40.592 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:40.594+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:40.710 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:40.713+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:40.841 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:40.844+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:41.621 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:41.625+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:41.732 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:41.736+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:41.796 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:41.798+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:42.650 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:42.653+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:42.702 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:42.705+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:42.759 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:42.763+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:43.636 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:43.639+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:43.665 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:43.668+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:43.712 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:43.716+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:44.644 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:44.647+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:44.650 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:44.654+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:44.701 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:44.704+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:45.610 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:45.614+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:45.630 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:45.633+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:45.748 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:45.752+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:46.640 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:46.644+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:46.660 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:46.663+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:46.759 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:46.763+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:47.630 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:47.634+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:47.669 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:47.673+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:47.745 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:47.748+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:48.635 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:48.639+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:48.692 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:48.696+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:48.704 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:48.708+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:49.653 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:49.656+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:49.662 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:49.666+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:49.703 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:49.707+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:50.666 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:50.670+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:50.701 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:50.704+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:50.724 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:50.727+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:51.682 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:51.685+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:51.694 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:51.697+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:51.715 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:51.719+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:52.692 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:52.696+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:52.729 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:52.733+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:52.738 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:52.742+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:53.695 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:53.698+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:53.728 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:53.732+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:53.743 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:53.746+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:54.680 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:54.682+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:54.704 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:54.707+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:54.763 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:54.765+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:55.723 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:55.725+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:55.724 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:55.726+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:55.724 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:55.727+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:56.711 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:56.714+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:56.712 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:56.715+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:56.757 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:56.760+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:57.665 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:57.668+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:57.667 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:57.670+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:57.804 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:57.806+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:58.620 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:58.622+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:58.627 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:58.630+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:58.830 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:58.833+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:59.607 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:03:59.609+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:59.620 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:03:59.623+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:03:59.841 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:03:59.844+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:00.624 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:00.626+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:00.624 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:00.627+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:00.807 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:00.810+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:01.618 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:01.621+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:01.671 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:01.674+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:01.791 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:01.793+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:02.578 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:02.581+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:02.704 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:02.706+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:02.768 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:02.770+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:03.607 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:03.610+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:03.711 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:03.713+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:03.816 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:03.819+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:04.610 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:04.612+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:04.681 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:04.683+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:04.857 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:04.860+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:05.607 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:05.609+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:05.699 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:05.702+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:05.904 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:05.906+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:06.640 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:06.643+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:06.746 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:06.749+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:06.912 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:06.915+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:07.592 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:07.594+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:07.743 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:07.746+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:07.918 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:07.921+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:08.582 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:08.585+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:08.751 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:08.753+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:08.930 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:08.933+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:09.630 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:09.632+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:09.776 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:09.779+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:09.963 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:09.966+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:10.632 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:10.635+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:10.750 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:10.752+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:10.980 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:10.983+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:11.594 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:11.597+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:11.776 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:11.779+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:11.982 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:11.981+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:12.558 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:12.561+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:12.824 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:12.827+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:12.940 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:12.943+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:13.580 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:13.583+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:13.817 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:13.819+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:13.915 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:13.918+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:14.593 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:14.596+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:14.820 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:14.822+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:14.944 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:14.947+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:15.606 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:15.609+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:15.827 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:15.830+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:15.994 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:15.997+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:16.573 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:16.575+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:16.857 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:16.860+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:16.981 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:16.984+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:17.567 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:17.570+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:17.854 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:17.857+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:17.992 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:17.995+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:18.566 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:18.568+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:18.899 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:18.902+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:18.997 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:19.000+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:19.575 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:19.578+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:19.929 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:19.931+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:20.034 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:20.036+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:20.538 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:20.540+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:20.955 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:20.958+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:21.040 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:21.043+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:21.510 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:21.513+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:21.929 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:21.932+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:22.044 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:22.047+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:22.554 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:22.557+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:22.886 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:22.889+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:23.005 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:23.007+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:23.507 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:23.510+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:23.847 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:23.850+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:23.964 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:23.966+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:24.550 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:24.553+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:24.856 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:24.860+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:24.983 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:24.986+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:25.561 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:25.564+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:25.843 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:25.846+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:25.940 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:25.944+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:26.608 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:26.611+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:26.861 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:26.864+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:26.903 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:26.907+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:27.561 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:27.565+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:27.820 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:27.823+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:27.857 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:27.861+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:28.605 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:28.609+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:28.865 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:28.869+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:28.906 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:28.909+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:29.638 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:29.641+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:29.865 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:29.869+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:29.892 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:29.896+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:30.649 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:30.652+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:30.851 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:30.855+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:30.879 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:30.883+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:31.692 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:31.695+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:31.862 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:31.865+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:31.866 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:31.870+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:32.653 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:32.657+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:32.869 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:32.873+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:32.884 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:32.888+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:33.701 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:33.704+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:33.835 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:33.839+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:33.927 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:33.931+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:34.726 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:34.729+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:34.852 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:34.855+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:34.883 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:34.887+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:35.707 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:35.710+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:35.889 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:35.893+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:35.930 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:35.934+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:36.741 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:36.744+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:36.849 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:36.852+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:36.957 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:36.961+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:37.752 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:37.755+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:37.834 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:37.837+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:37.974 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:37.977+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:38.784 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:38.787+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:38.811 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:38.814+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:38.981 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:38.983+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:39.777 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:39.780+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:39.784 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:39.787+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:39.990 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:39.993+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:40.755 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:40.758+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:40.763 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:40.766+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:40.948 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:40.951+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:41.737 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:41.740+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:41.776 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:41.778+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:41.954 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:41.956+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:42.727 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:42.729+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:42.780 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:42.783+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:42.943 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:42.945+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:43.761 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:43.764+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:43.763 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:43.766+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:43.913 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:43.916+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:44.779 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:44.782+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:44.790 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:44.793+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:44.961 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:44.963+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:45.764 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:45.767+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:45.769 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:45.772+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:45.974 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:45.977+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:46.788 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:46.791+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:46.809 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:46.812+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:46.996 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:46.999+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:47.747 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:47.750+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:47.763 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:47.766+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:48.039 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:48.042+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:48.760 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:48.763+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:48.793 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:48.795+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:49.084 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:49.087+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:49.746 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:49.749+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:49.822 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:49.824+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:50.090 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:50.092+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:50.743 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:50.746+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:50.865 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:50.868+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:51.138 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:51.140+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:51.719 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:51.722+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:51.883 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:51.886+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:52.167 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:52.170+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:52.748 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:52.751+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:52.912 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:52.914+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:53.132 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:53.134+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:53.748 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:53.750+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:53.960 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:53.963+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:54.107 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:54.110+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:54.777 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:54.780+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:54.930 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:54.932+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:55.089 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:55.091+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:55.813 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:55.816+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:55.931 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:55.934+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:56.061 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:56.064+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:56.811 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:56.814+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:56.887 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:56.890+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:57.093 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:57.095+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:57.779 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:57.781+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:57.889 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:57.892+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:58.084 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:58.086+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:58.805 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:58.808+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:58.843 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:58.846+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:59.090 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:04:59.093+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:59.854 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:04:59.856+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:04:59.854 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:04:59.856+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:00.137 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:00.139+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:00.888 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:00.891+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:00.889 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:00.891+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:01.150 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:01.153+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:01.855 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:01.858+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:01.933 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:01.936+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:02.162 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:02.165+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:02.812 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:02.815+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:02.949 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:02.952+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:03.174 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:03.177+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:03.807 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:03.809+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:03.936 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:03.938+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:04.216 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:04.219+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:04.761 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:04.763+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:04.940 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:04.943+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:05.225 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:05.227+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:05.798 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:05.800+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:05.985 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:05.986+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:06.212 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:06.214+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:06.829 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:06.832+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:06.999 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:07.002+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:07.257 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:07.259+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:07.815 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:07.817+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:07.959 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:07.962+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:08.265 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:08.267+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:08.853 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:08.855+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:08.997 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:08.999+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:09.225 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:09.228+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:09.893 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:09.895+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:10.038 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:10.040+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:10.205 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:10.208+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:10.864 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:10.868+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:11.019 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:11.023+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:11.229 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:11.232+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:11.829 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:11.833+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:12.009 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:12.012+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:12.256 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:12.260+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:12.841 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:12.845+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:13.058 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:13.061+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:13.258 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:13.261+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:13.801 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:13.805+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:14.089 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:14.093+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:14.271 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:14.275+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:14.828 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:14.832+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:15.125 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:15.129+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:15.243 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:15.247+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:15.801 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:15.804+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:16.131 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:16.135+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:16.290 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:16.293+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:16.835 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:16.839+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:17.167 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:17.171+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:17.266 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:17.270+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:17.883 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:17.887+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:18.214 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:18.218+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:18.247 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:18.251+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:18.871 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:18.875+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:19.219 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:19.222+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:19.255 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:19.259+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:19.869 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:19.873+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:20.192 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:20.195+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:20.220 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:20.224+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:20.845 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:20.848+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:21.209 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:21.213+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:21.227 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:21.230+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:21.858 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:21.861+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:22.215 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:22.218+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:22.268 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:22.272+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:22.877 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:22.881+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:23.193 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:23.196+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:23.260 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:23.262+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:23.903 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:23.905+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:24.161 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:24.163+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:24.224 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:24.226+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:24.922 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:24.925+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:25.115 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:25.118+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:25.272 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:25.275+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:25.933 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:25.936+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:26.139 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:26.141+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:26.292 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:26.295+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:26.930 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:26.932+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:27.120 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:27.122+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:27.306 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:27.309+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:27.924 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:27.927+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:28.134 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:28.137+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:28.304 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:28.307+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:28.947 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:28.950+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:29.095 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:29.098+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:29.305 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:29.308+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:29.976 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:29.978+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:30.056 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:30.059+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:30.292 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:30.295+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:31.021 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:31.024+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:31.066 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:31.068+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:31.290 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:31.293+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:31.992 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:31.995+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:32.071 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:32.074+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:32.270 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:32.273+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:32.977 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:32.980+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:33.078 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:33.081+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:33.240 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:33.243+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:33.950 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:33.952+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:34.045 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:34.047+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:34.258 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:34.261+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:34.916 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:34.919+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:35.057 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:35.059+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:35.219 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:35.221+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:35.872 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:35.875+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:36.037 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:36.040+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:36.209 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:36.211+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:36.851 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:36.854+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:37.063 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:37.065+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:37.243 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:37.246+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:37.878 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:37.880+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:38.051 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:38.054+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:38.239 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:38.242+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:38.911 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:38.914+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:39.079 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:39.081+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:39.208 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:39.211+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:39.892 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:39.894+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:40.107 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:40.109+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:40.247 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:40.250+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:40.911 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:40.914+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:41.077 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:41.080+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:41.210 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:41.212+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:41.911 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:41.913+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:42.041 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:42.043+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:42.190 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:42.193+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:42.934 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:42.937+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:42.998 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:43.000+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:43.229 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:43.231+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:43.886 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:43.888+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:44.038 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:44.040+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:44.268 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:44.271+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:44.875 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:44.878+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:45.023 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:45.026+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:45.297 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:45.300+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:45.873 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:45.876+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:45.993 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:45.995+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:46.272 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:46.275+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:46.871 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:46.873+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:47.010 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:47.012+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:47.222 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:47.225+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:47.826 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:47.829+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:48.006 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:48.008+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:48.256 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:48.259+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:48.789 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:48.791+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:49.022 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:49.024+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:49.224 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:49.227+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:49.761 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:49.764+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:49.991 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:49.994+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:50.188 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:50.191+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:50.777 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:50.780+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:51.007 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:51.010+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:51.196 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:51.199+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:51.743 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:51.746+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:52.043 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:52.046+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:52.185 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:52.188+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:52.695 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:52.698+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:53.039 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:53.041+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:53.191 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:53.195+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:53.664 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:53.667+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:54.009 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:54.013+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:54.208 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:54.212+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:54.673 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:54.676+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:55.031 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:55.035+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:55.248 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:55.252+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:55.640 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:55.644+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:56.023 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:56.027+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:56.218 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:56.221+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:56.604 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:56.608+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:57.059 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:57.063+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:57.259 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:57.263+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:57.574 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:57.577+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:58.037 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:58.041+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:58.241 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:58.245+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:58.529 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:58.533+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:59.028 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:05:59.032+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:59.198 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:05:59.202+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:05:59.509 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:05:59.513+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:00.065 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:00.069+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:00.157 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:00.161+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:00.470 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:00.474+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:01.077 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:01.081+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:01.199 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:01.203+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:01.460 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:01.463+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:02.123 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:02.127+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:02.188 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:02.192+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:02.483 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:02.487+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:03.164 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:03.168+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:03.228 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:03.232+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:03.492 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:03.496+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:04.137 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:04.141+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:04.223 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:04.227+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:04.449 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:04.453+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:05.095 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:05.099+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:05.253 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:05.256+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:05.477 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:05.481+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:06.079 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:06.083+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:06.208 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:06.212+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:06.445 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:06.449+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:07.112 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:07.115+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:07.182 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:07.185+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:07.425 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:07.429+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:08.120 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:08.123+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:08.143 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:08.146+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:08.409 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:08.412+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:09.105 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:09.107+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:09.136 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:09.139+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:09.412 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:09.415+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:10.072 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:10.075+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:10.095 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:10.098+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:10.371 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:10.374+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:11.047 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:11.050+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:11.105 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:11.106+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:11.411 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:11.414+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:12.061 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:12.064+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:12.084 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:12.087+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:12.372 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:12.375+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:13.027 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:13.029+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:13.120 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:13.122+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:13.373 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:13.376+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:14.018 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:14.021+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:14.087 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:14.090+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:14.341 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:14.344+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:15.053 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:15.056+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:15.057 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:15.060+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:15.353 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:15.356+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:16.049 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:16.052+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:16.084 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:16.087+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:16.328 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:16.331+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:17.029 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:17.032+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:17.061 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:17.064+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:17.338 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:17.340+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:17.994 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:17.997+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:18.093 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:18.096+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:18.306 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:18.308+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:19.024 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:19.027+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:19.055 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:19.058+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:19.306 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:19.309+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:20.068 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:20.071+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:20.085 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:20.088+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:20.341 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:20.344+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:21.025 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:21.027+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:21.065 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:21.068+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:21.326 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:21.328+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:22.019 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:22.021+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:22.089 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:22.092+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:22.301 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:22.304+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:22.985 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:22.987+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:23.057 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:23.060+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:23.263 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:23.266+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:24.027 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:24.029+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:24.059 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:24.062+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:24.240 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:24.242+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:25.013 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:25.016+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:25.060 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:25.063+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:25.256 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:25.258+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:26.034 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:26.036+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:26.081 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:26.083+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:26.297 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:26.300+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:27.055 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:27.057+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:27.094 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:27.096+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:27.273 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:27.276+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:28.079 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:28.082+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:28.084 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:28.087+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:28.237 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:28.240+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:29.055 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:29.058+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:29.095 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:29.098+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:29.209 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:29.212+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:30.073 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:30.076+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:30.139 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:30.141+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:30.239 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:30.242+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:31.083 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:31.086+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:31.152 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:31.155+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:31.198 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:31.201+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:32.119 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:32.121+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:32.158 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:32.161+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:32.239 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:32.242+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:33.125 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:33.127+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:33.157 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:33.160+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:33.213 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:33.216+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:34.101 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:34.103+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:34.200 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:34.202+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:34.203 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:34.206+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:35.120 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:35.122+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:35.157 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:35.160+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:35.218 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:35.221+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:36.151 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:36.154+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:36.168 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:36.171+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:36.229 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:36.231+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:37.111 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:37.113+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:37.143 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:37.146+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:37.195 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:37.198+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:38.123 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:38.126+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:38.183 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:38.186+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:38.231 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:38.233+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:39.135 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:39.139+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:39.141 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:39.145+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:39.198 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:39.202+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:40.146 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:40.149+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:40.163 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:40.167+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:40.215 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:40.219+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:41.163 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:41.167+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:41.167 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:41.171+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:41.179 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:41.183+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:42.165 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:42.168+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:42.191 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:42.195+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:42.202 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:42.205+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:43.170 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:43.174+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:43.186 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:43.190+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:43.224 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:43.228+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:44.139 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:44.143+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:44.153 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:44.157+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:44.226 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:44.230+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:45.159 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:45.163+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:45.168 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:45.172+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:45.257 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:45.261+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:46.121 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:46.125+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:46.219 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:46.223+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:46.256 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:46.260+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:47.081 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:47.085+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:47.204 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:47.207+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:47.293 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:47.296+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:48.112 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:48.116+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:48.232 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:48.235+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:48.301 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:48.305+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:49.078 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:49.081+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:49.237 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:49.241+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:49.310 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:49.313+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:50.095 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:50.099+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:50.273 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:50.277+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:50.329 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:50.333+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:51.097 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:51.100+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:51.277 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:51.278+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:51.296 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:51.300+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:52.121 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:52.124+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:52.253 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:52.256+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:52.262 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:52.265+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:53.084 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:53.086+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:53.250 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:53.252+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:53.269 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:53.272+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:54.093 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:54.095+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:54.251 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:54.254+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:54.275 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:54.278+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:55.063 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:55.066+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:55.254 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:55.257+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:55.264 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:55.267+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:56.105 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:56.108+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:56.251 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:56.254+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:56.306 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:56.309+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:57.068 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:57.071+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:57.252 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:57.255+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:57.261 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:57.264+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:58.075 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:58.078+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:58.233 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:58.236+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:58.268 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:58.270+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:59.122 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:06:59.124+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:59.216 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:06:59.219+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:06:59.297 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:06:59.300+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:00.101 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:00.103+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:00.223 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:00.226+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:00.325 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:00.328+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:01.053 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:01.055+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:01.250 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:01.253+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:01.278 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:01.280+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:02.039 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:02.042+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:02.246 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:02.249+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:02.251 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:02.254+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:03.069 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:03.072+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:03.203 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:03.206+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:03.216 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:03.219+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:04.052 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:04.055+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:04.172 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:04.175+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:04.243 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:04.245+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:05.029 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:05.031+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:05.126 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:05.129+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:05.276 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:05.279+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:05.991 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:05.993+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:06.158 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:06.161+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:06.252 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:06.254+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:06.998 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:07.001+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:07.157 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:07.160+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:07.238 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:07.241+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:08.006 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:08.008+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:08.121 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:08.124+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:08.280 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:08.282+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:09.033 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:09.035+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:09.103 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:09.106+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:09.248 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:09.250+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:09.988 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:09.991+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:10.148 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:10.151+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:10.252 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:10.255+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:10.997 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:11.000+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:11.148 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:11.151+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:11.266 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:11.269+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:11.959 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:11.962+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:12.185 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:12.188+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:12.231 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:12.234+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:12.943 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:12.946+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:13.208 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:13.211+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:13.230 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:13.233+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:13.972 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:13.974+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:14.163 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:14.166+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:14.184 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:14.187+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:14.997 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:14.999+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:15.181 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:15.184+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:15.210 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:15.212+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:15.980 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:15.983+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:16.137 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:16.139+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:16.171 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:16.174+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:16.987 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:16.990+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:17.170 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:17.173+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:17.213 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:17.215+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:17.964 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:17.967+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:18.153 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:18.156+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:18.261 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:18.263+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:18.919 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:18.921+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:19.115 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:19.118+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:19.304 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:19.307+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:19.901 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:19.904+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:20.101 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:20.104+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:20.332 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:20.335+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:20.917 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:20.919+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:21.134 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:21.137+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:21.366 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:21.369+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:21.945 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:21.948+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:22.089 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:22.092+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:22.321 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:22.324+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:22.923 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:22.926+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:23.099 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:23.101+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:23.288 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:23.290+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:23.931 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:23.934+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:24.085 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:24.088+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:24.330 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:24.334+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:24.966 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:24.970+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:25.068 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:25.072+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:25.303 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:25.307+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:25.917 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:25.921+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:26.119 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:26.122+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:26.254 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:26.257+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:26.949 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:26.953+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:27.157 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:27.161+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:27.298 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:27.302+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:27.995 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:27.998+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:28.127 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:28.131+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:28.272 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:28.275+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:29.007 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:29.011+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:29.156 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:29.160+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:29.281 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:29.285+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:29.989 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:29.993+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:30.129 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:30.132+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:30.319 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:30.323+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:30.965 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:30.969+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:31.091 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:31.094+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:31.314 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:31.317+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:31.924 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:31.927+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:32.060 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:32.064+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:32.288 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:32.292+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:32.929 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:32.932+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:33.085 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:33.089+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:33.308 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:33.312+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:33.887 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:33.890+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:34.089 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:34.093+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:34.261 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:34.265+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:34.863 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:34.867+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:35.115 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:35.119+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:35.273 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:35.276+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:35.843 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:35.847+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:36.161 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:36.165+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:36.291 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:36.294+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:36.869 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:36.872+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:37.183 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:37.186+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:37.272 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:37.275+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:37.894 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:37.896+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:38.225 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:38.228+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:38.234 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:38.237+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:38.917 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:38.919+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:39.210 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:39.212+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:39.231 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:39.234+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:39.899 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:39.902+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:40.219 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:40.221+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:40.219 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:40.222+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:40.933 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:40.936+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:41.255 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:41.258+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:41.261 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:41.264+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:41.902 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:41.904+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:42.215 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:42.218+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:42.277 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:42.277+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:42.878 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:42.880+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:43.180 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:43.183+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:43.283 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:43.286+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:43.849 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:43.852+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:44.187 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:44.189+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:44.307 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:44.310+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:44.811 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:44.814+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:45.165 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:45.167+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:45.319 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:45.322+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:45.815 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:45.818+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:46.121 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:46.124+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:46.305 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:46.308+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:46.791 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:46.794+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:47.144 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:47.147+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:47.277 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:47.280+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:47.747 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:47.749+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:48.143 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:48.145+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:48.249 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:48.252+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:48.730 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:48.733+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:49.132 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:49.135+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:49.212 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:49.215+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:49.708 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:49.710+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:50.121 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:50.123+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:50.223 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:50.226+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:50.680 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:50.683+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:51.114 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:51.117+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:51.248 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:51.251+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:51.723 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:51.726+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:52.139 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:52.141+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:52.228 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:52.231+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:52.735 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:52.738+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:53.179 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:53.181+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:53.214 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:53.217+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:53.749 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:53.752+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:54.221 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:54.224+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:54.227 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:54.230+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:54.728 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:54.731+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:55.172 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:55.175+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:55.225 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:55.228+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:55.684 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:55.687+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:56.144 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:56.147+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:56.252 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:56.255+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:56.645 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:56.648+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:57.151 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:57.154+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:57.259 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:57.262+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:57.644 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:57.646+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:58.144 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:58.147+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:58.224 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:58.227+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:58.598 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:58.600+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:59.103 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:07:59.105+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:59.247 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:07:59.250+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:07:59.589 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:07:59.591+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:00.072 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:00.074+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:00.236 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:00.239+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:00.620 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:00.622+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:01.108 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:01.111+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:01.190 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:01.193+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:01.625 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:01.627+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:02.152 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:02.154+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:02.194 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:02.196+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:02.587 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:02.590+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:03.178 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:03.181+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:03.242 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:03.244+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:03.555 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:03.558+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:04.193 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:04.196+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:04.217 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:04.220+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:04.569 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:04.572+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:05.176 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:05.179+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:05.237 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:05.240+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:05.618 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:05.621+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:06.182 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:06.186+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:06.192 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:06.195+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:06.630 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:06.632+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:07.143 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:07.146+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:07.175 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:07.177+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:07.671 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:07.674+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:08.185 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:08.189+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:08.216 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:08.220+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:08.690 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:08.693+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:09.166 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:09.170+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:09.182 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:09.185+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:09.735 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:09.738+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:10.201 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:10.204+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:10.225 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:10.229+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:10.745 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:10.749+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:11.198 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:11.202+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:11.266 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:11.269+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:11.702 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:11.706+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:12.164 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:12.168+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:12.238 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:12.242+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:12.670 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:12.674+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:13.166 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:13.170+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:13.237 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:13.241+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:13.641 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:13.645+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:14.212 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:14.216+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:14.243 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:14.247+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:14.634 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:14.636+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:15.225 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:15.229+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:15.250 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:15.254+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:15.609 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:15.612+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:16.197 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:16.200+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:16.211 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:16.214+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:16.599 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:16.603+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:17.212 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:17.215+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:17.246 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:17.250+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:17.609 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:17.613+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:18.204 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:18.207+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:18.208 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:18.212+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:18.657 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:18.661+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:19.190 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:19.193+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:19.219 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:19.222+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:19.666 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:19.669+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:20.238 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:20.241+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:20.238 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:20.242+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:20.706 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:20.710+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:21.245 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:21.248+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:21.260 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:21.263+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:21.667 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:21.669+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:22.289 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:22.292+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:22.294 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:22.297+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:22.674 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:22.677+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:23.246 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:23.249+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:23.327 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:23.330+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:23.646 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:23.649+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:24.282 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:24.284+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:24.285 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:24.288+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:24.688 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:24.691+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:25.279 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:25.282+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:25.322 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:25.325+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:25.666 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:25.668+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:26.282 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:26.285+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:26.291 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:26.294+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:26.706 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:26.709+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:27.289 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:27.292+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:27.336 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:27.338+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:27.743 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:27.745+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:28.258 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:28.261+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:28.330 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:28.332+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:28.707 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:28.710+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:29.211 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:29.213+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:29.321 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:29.324+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:29.723 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:29.725+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:30.245 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:30.248+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:30.354 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:30.357+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:30.739 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:30.742+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:31.203 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:31.206+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:31.342 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:31.345+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:31.702 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:31.705+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:32.161 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:32.164+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:32.323 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:32.326+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:32.660 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:32.663+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:33.188 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:33.191+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:33.348 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:33.351+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:33.708 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:33.711+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:34.139 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:34.141+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:34.353 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:34.356+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:34.662 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:34.664+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:35.186 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:35.189+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:35.331 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:35.334+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:35.712 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:35.715+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:36.223 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:36.226+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:36.298 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:36.301+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:36.692 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:36.695+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:37.206 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:37.209+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:37.274 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:37.276+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:37.661 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:37.664+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:38.180 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:38.183+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:38.260 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:38.263+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:38.636 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:38.639+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:39.158 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:39.160+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:39.248 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:39.249+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:39.676 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:39.678+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:40.181 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:40.184+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:40.204 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:40.207+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:40.636 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:40.639+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:41.193 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:41.195+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:41.206 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:41.208+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:41.663 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:41.666+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:42.221 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:42.223+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:42.253 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:42.255+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:42.630 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:42.633+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:43.202 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:43.205+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:43.206 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:43.209+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:43.640 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:43.643+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:44.235 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:44.238+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:44.253 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:44.256+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:44.666 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:44.669+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:45.200 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:45.203+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:45.210 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:45.213+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:45.707 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:45.710+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:46.171 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:46.174+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:46.207 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:46.209+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:46.747 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:46.750+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:47.179 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:47.181+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:47.222 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:47.225+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:47.736 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:47.738+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:48.158 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:48.160+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:48.180 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:48.183+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:48.774 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:48.776+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:49.108 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:49.111+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:49.151 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:49.154+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:49.787 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:49.789+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:50.059 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:50.061+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:50.146 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:50.149+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:50.782 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:50.784+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:51.082 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:51.086+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:51.138 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:51.140+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:51.755 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:51.758+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:52.095 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:52.099+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:52.109 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:52.112+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:52.747 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:52.751+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:53.121 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:53.125+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:53.130 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:53.134+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:53.700 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:53.704+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:54.109 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:54.113+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:54.160 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:54.164+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:54.705 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:54.709+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:55.090 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:55.094+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:55.112 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:55.116+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:55.745 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:55.749+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:56.072 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:56.074+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:56.083 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:56.087+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:56.739 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:56.743+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:57.057 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:57.061+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:57.059 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:57.063+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:57.784 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:57.787+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:58.014 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:58.018+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:58.043 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:58.046+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:58.795 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:58.798+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:59.053 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:08:59.057+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:59.084 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:08:59.088+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:08:59.841 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:08:59.845+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:00.099 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:00.103+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:00.107 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:00.111+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:00.832 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:00.835+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:01.081 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:01.084+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:01.102 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:01.106+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:01.839 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:01.843+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:02.049 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:02.053+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:02.080 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:02.084+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:02.800 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:02.804+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:03.033 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:03.037+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:03.066 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:03.070+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:03.820 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:03.823+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:04.017 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:04.020+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:04.035 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:04.039+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:04.842 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:04.846+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:04.995 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:04.999+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:05.064 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:05.068+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:05.842 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:05.844+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:06.001 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:06.003+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:06.017 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:06.020+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:06.863 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:06.866+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:06.956 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:06.959+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:06.973 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:06.975+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:07.830 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:07.833+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:07.969 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:07.971+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:07.991 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:07.994+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:08.857 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:08.860+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:08.970 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:08.973+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:08.974 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:08.977+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:09.879 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:09.882+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:09.967 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:09.969+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:09.987 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:09.990+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:10.907 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:10.910+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:10.947 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:10.949+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:10.981 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:10.983+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:11.897 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:11.899+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:11.935 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:11.938+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:11.984 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:11.986+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:12.892 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:12.895+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:12.935 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:12.937+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:12.973 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:12.975+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:13.908 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:13.910+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:13.926 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:13.929+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:13.975 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:13.978+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:14.873 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:14.876+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:14.894 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:14.896+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:14.947 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:14.949+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:15.860 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:15.862+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:15.860 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:15.863+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:15.974 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:15.977+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:16.839 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:16.841+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:16.884 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:16.886+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:17.009 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:17.012+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:17.809 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:17.808+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:17.901 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:17.904+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:18.016 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:18.018+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:18.757 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:18.759+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:18.883 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:18.886+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:19.048 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:19.050+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:19.767 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:19.770+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:19.915 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:19.918+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:20.085 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:20.088+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:20.780 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:20.783+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:20.937 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:20.940+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:21.113 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:21.116+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:21.796 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:21.799+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:21.920 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:21.922+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:22.106 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:22.109+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:22.783 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:22.786+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:22.951 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:22.953+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:23.139 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:23.141+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:23.752 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:23.755+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:23.922 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:23.924+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:24.095 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:24.098+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:24.782 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:24.785+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:24.957 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:24.960+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:25.082 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:25.084+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:25.786 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:25.789+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:26.002 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:26.004+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:26.109 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:26.112+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:26.744 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:26.746+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:26.971 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:26.974+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:27.123 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:27.125+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:27.714 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:27.716+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:27.989 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:27.991+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:28.148 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:28.150+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:28.716 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:28.719+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:29.023 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:29.025+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:29.113 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:29.116+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:29.683 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:29.685+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:30.017 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:30.020+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:30.086 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:30.088+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:30.643 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:30.645+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:31.037 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:31.040+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:31.111 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:31.114+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:31.623 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:31.625+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:32.068 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:32.070+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:32.091 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:32.094+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:32.575 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:32.577+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:33.042 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:33.045+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:33.043 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:33.045+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:33.596 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:33.598+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:34.032 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:34.034+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:34.081 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:34.084+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:34.616 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:34.620+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:35.050 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:35.053+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:35.078 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:35.081+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:35.609 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:35.612+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:36.037 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:36.040+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:36.053 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:36.056+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:36.639 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:36.642+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:37.020 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:37.023+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:37.053 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:37.056+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:37.673 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:37.677+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:38.008 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:38.012+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:38.038 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:38.041+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:38.662 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:38.665+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:38.993 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:38.997+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:39.058 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:39.061+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:39.645 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:39.648+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:39.949 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:39.953+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:40.059 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:40.063+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:40.617 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:40.621+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:40.905 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:40.909+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:41.097 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:41.101+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:41.590 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:41.594+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:41.953 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:41.956+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:42.137 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:42.140+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:42.607 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:42.611+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:42.958 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:42.962+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:43.177 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:43.180+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:43.654 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:43.658+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:43.915 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:43.919+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:44.139 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:44.143+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:44.647 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:44.650+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:44.923 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:44.927+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:45.184 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:45.187+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:45.684 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:45.688+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:45.971 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:45.975+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:46.139 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:46.143+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:46.730 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:46.731+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:46.934 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:46.934+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:47.190 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:47.193+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:47.682 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:47.686+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:47.950 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:47.954+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:48.214 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:48.218+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:48.672 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:48.675+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:48.982 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:48.985+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:49.223 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:49.227+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:49.628 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:49.631+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:49.972 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:49.975+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:50.195 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:50.198+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:50.616 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:50.619+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:51.021 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:51.022+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:51.218 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:51.221+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:51.640 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:51.643+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:51.983 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:51.986+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:52.207 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:52.210+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:52.661 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:52.662+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:52.979 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:52.981+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:53.160 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:53.163+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:53.652 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:53.655+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:54.023 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:54.025+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:54.185 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:54.187+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:54.693 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:54.696+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:55.042 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:55.044+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:55.188 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:55.191+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:55.652 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:55.655+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:56.063 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:56.065+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:56.173 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:56.176+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:56.624 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:56.626+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:57.047 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:57.050+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:57.193 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:57.196+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:57.664 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:57.667+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:58.061 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:58.064+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:58.203 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:58.206+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:58.677 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:58.679+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:59.041 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:09:59.044+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:59.215 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:09:59.218+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:09:59.706 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:09:59.709+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:00.028 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:00.030+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:00.250 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:00.252+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:00.661 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:00.664+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:01.036 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:01.038+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:01.246 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:01.249+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:01.653 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:01.655+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:02.023 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:02.026+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:02.279 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:02.281+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:02.680 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:02.683+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:03.010 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:03.012+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:03.261 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:03.264+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:03.640 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:03.643+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:04.029 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:04.032+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:04.283 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:04.286+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:04.602 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:04.605+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:04.993 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:04.996+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:05.235 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:05.237+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:05.619 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:05.621+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:06.013 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:06.016+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:06.203 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:06.206+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:06.603 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:06.606+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:06.999 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:07.002+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:07.162 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:07.164+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:07.554 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:07.557+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:08.047 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:08.050+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:08.165 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:08.168+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:08.602 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:08.604+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:09.093 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:09.096+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:09.168 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:09.170+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:09.577 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:09.579+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:10.097 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:10.099+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:10.179 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:10.181+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:10.605 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:10.608+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:11.116 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:11.118+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:11.217 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:11.220+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:11.646 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:11.648+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:12.156 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:12.159+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:12.238 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:12.240+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:12.624 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:12.626+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:13.182 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:13.184+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:13.221 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:13.224+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:13.599 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:13.601+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:14.173 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:14.175+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:14.215 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:14.218+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:14.645 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:14.647+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:15.201 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:15.204+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:15.261 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:15.263+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:15.690 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:15.693+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:16.229 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:16.231+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:16.245 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:16.247+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:16.688 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:16.690+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:17.224 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:17.226+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:17.285 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:17.287+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:17.696 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:17.699+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:18.233 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:18.236+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:18.333 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:18.336+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:18.668 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:18.670+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:19.211 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:19.214+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:19.346 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:19.348+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:19.676 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:19.679+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:20.232 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:20.235+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:20.387 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:20.390+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:20.719 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:20.722+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:21.219 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:21.222+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:21.384 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:21.386+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:21.672 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:21.674+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:22.244 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:22.247+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:22.370 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:22.374+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:22.720 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:22.723+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:23.264 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:23.267+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:23.330 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:23.334+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:23.755 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:23.759+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:24.241 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:24.245+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:24.293 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:24.297+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:24.747 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:24.751+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:25.206 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:25.210+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:25.304 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:25.308+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:25.793 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:25.797+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:26.232 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:26.235+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:26.295 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:26.298+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:26.744 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:26.748+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:27.232 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:27.236+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:27.298 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:27.301+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:27.753 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:27.757+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:28.190 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:28.193+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:28.258 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:28.262+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:28.749 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:28.752+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:29.208 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:29.212+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:29.245 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:29.248+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:29.728 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:29.731+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:30.163 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:30.166+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:30.289 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:30.292+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:30.761 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:30.764+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:31.185 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:31.188+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:31.264 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:31.267+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:31.766 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:31.769+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:32.156 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:32.160+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:32.293 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:32.296+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:32.757 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:32.761+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:33.117 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:33.120+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:33.326 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:33.330+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:33.727 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:33.730+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:34.069 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:34.072+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:34.292 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:34.296+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:34.698 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:34.702+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:35.090 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:35.093+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:35.296 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:35.298+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:35.743 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:35.746+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:36.075 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:36.077+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:36.327 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:36.329+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:36.763 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:36.765+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:37.032 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:37.035+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:37.279 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:37.282+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:37.790 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:37.792+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:38.082 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:38.084+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:38.310 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:38.313+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:38.824 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:38.826+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:39.073 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:39.075+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:39.262 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:39.265+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:39.838 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:39.841+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:40.047 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:40.049+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:40.226 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:40.228+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:40.864 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:40.866+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:41.093 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:41.095+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:41.190 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:41.192+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:41.884 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:41.887+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:42.143 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:42.145+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:42.194 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:42.197+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:42.901 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:42.904+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:43.150 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:43.152+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:43.164 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:43.166+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:43.878 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:43.880+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:44.134 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:44.136+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:44.159 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:44.161+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:44.859 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:44.861+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:45.140 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:45.143+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:45.163 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:45.165+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:45.882 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:45.884+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:46.145 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:46.147+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:46.146 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:46.149+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:46.845 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:46.848+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:47.170 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:47.173+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:47.190 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:47.192+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:47.822 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:47.824+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:48.155 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:48.156+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:48.170 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:48.172+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:48.820 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:48.823+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:49.191 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:49.193+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:49.221 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:49.223+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:49.803 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:49.804+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:50.177 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:50.179+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:50.207 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:50.209+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:50.775 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:50.778+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:51.190 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:51.193+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:51.204 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:51.206+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:51.731 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:51.733+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:52.148 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:52.150+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:52.216 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:52.218+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:52.778 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:52.781+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:53.143 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:53.145+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:53.194 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:53.196+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:53.823 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:53.826+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:54.097 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:54.099+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:54.194 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:54.196+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:54.789 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:54.791+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:55.095 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:55.097+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:55.196 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:55.199+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:55.771 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:55.773+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:56.095 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:56.097+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:56.239 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:56.241+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:56.818 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:56.820+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:57.077 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:57.077+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:57.238 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:57.240+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:57.829 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:57.831+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:58.117 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:58.119+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:58.222 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:58.225+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:58.788 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:58.790+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:59.138 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:10:59.140+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:59.240 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:10:59.243+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:10:59.823 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:10:59.825+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:00.141 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:00.143+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:00.251 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:00.253+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:00.830 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:00.833+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:01.103 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:01.106+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:01.225 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:01.227+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:01.840 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:01.842+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:02.089 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:02.091+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:02.227 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:02.229+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:02.800 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:02.802+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:03.061 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:03.063+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:03.215 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:03.217+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:03.798 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:03.801+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:04.041 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:04.043+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:04.188 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:04.190+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:04.765 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:04.767+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:05.056 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:05.058+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:05.217 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:05.219+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:05.728 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:05.731+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:06.099 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:06.102+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:06.249 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:06.252+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:06.681 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:06.684+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:07.052 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:07.055+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:07.270 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:07.273+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:07.641 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:07.643+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:08.032 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:08.035+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:08.238 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:08.241+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:08.667 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:08.670+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:09.031 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:09.035+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:09.242 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:09.245+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:09.645 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:09.648+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:10.010 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:10.013+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:10.199 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:10.202+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:10.624 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:10.627+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:10.985 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:10.989+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:11.150 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:11.153+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:11.654 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:11.657+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:11.939 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:11.942+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:12.124 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:12.127+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:12.688 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:12.692+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:12.912 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:12.915+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:13.134 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:13.137+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:13.657 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:13.660+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:13.873 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:13.876+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:14.096 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:14.099+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:14.685 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:14.689+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:14.925 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:14.926+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:15.111 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:15.115+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:15.708 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:15.711+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:15.888 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:15.891+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:16.119 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:16.123+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:16.711 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:16.714+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:16.907 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:16.911+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:17.136 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:17.139+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:17.754 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:17.758+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:17.909 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:17.912+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:18.107 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:18.110+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:18.736 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:18.739+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:18.885 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:18.888+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:19.064 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:19.067+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:19.720 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:19.722+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:19.853 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:19.856+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:20.039 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:20.041+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:20.698 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:20.700+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:20.861 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:20.864+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:20.998 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:21.000+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:21.681 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:21.683+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:21.865 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:21.868+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:22.011 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:22.013+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:22.665 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:22.667+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:22.824 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:22.826+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:23.054 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:23.056+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:23.616 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:23.618+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:23.875 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:23.877+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:24.030 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:24.032+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:24.566 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:24.569+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:24.879 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:24.882+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:25.011 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:25.013+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:25.576 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:25.578+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:25.838 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:25.840+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:25.974 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:25.976+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:26.530 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:26.532+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:26.820 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:26.822+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:26.964 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:26.966+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:27.579 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:27.581+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:27.839 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:27.840+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:27.982 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:27.984+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:28.616 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:28.618+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:28.830 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:28.832+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:29.006 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:29.008+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:29.664 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:29.666+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:29.843 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:29.845+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:30.029 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:30.031+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:30.683 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:30.686+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:30.857 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:30.859+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:31.071 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:31.073+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:31.716 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:31.718+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:31.811 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:31.814+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:32.070 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:32.072+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:32.675 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:32.677+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:32.846 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:32.848+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:33.096 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:33.098+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:33.689 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:33.690+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:33.800 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:33.802+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:34.135 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:34.137+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:34.725 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:34.725+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:34.787 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:34.789+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:35.158 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:35.160+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:35.680 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:35.681+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:35.790 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:35.792+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:36.202 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:36.204+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:36.677 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:36.679+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:36.836 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:36.838+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:37.226 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:37.228+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:37.715 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:37.717+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:37.832 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:37.834+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:38.268 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:38.270+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:38.738 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:38.740+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:38.792 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:38.794+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:39.281 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:39.283+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:39.690 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:39.693+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:39.836 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:39.838+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:40.306 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:40.308+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:40.726 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:40.728+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:40.846 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:40.848+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:41.264 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:41.266+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:41.738 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:41.740+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:41.879 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:41.881+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:42.242 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:42.244+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:42.779 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:42.781+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:42.900 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:42.902+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:43.213 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:43.215+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:43.743 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:43.745+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:43.858 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:43.860+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:44.240 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:44.242+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:44.732 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:44.734+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:44.848 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:44.850+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:45.191 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:45.193+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:45.709 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:45.712+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:45.864 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:45.866+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:46.208 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:46.210+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:46.758 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:46.760+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:46.824 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:46.826+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:47.166 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:47.168+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:47.785 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:47.787+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:47.850 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:47.852+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:48.194 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:48.196+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:48.756 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:48.758+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:48.842 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:48.844+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:49.211 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:49.213+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:49.749 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:49.750+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:49.853 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:49.855+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:50.170 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:50.172+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:50.758 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:50.760+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:50.872 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:50.874+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:51.169 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:51.172+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:51.793 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:51.796+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:51.921 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:51.924+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:52.207 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:52.210+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:52.746 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:52.749+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:52.927 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:52.930+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:53.176 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:53.179+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:53.761 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:53.764+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:53.976 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:53.979+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:54.156 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:54.159+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:54.782 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:54.785+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:54.945 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:54.948+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:55.173 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:55.176+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:55.784 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:55.787+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:55.900 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:55.903+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:56.168 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:56.171+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:56.760 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:56.763+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:56.940 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:56.943+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:57.214 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:57.217+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:57.755 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:57.757+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:57.984 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:57.987+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:58.199 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:58.202+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:58.750 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:58.753+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:58.975 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:58.978+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:59.161 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:11:59.164+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:59.784 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:11:59.786+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:11:59.937 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:11:59.940+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:00.126 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:00.129+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:00.762 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:00.765+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:00.894 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:00.897+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:01.137 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:01.140+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:01.793 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:01.796+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:01.868 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:01.871+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:02.153 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:02.156+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:02.814 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:02.817+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:02.820 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:02.823+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:03.176 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:03.179+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:03.825 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:03.828+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:03.830 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:03.833+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:04.172 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:04.175+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:04.779 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:04.781+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:04.860 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:04.862+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:05.213 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:05.215+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:05.743 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:05.745+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:05.909 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:05.911+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:06.187 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:06.189+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:06.757 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:06.759+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:06.909 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:06.911+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:07.170 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:07.172+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:07.757 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:07.759+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:07.936 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:07.938+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:08.193 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:08.195+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:08.803 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:08.805+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:08.938 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:08.940+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:09.144 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:09.146+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:09.776 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:09.778+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:09.907 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:09.908+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:10.131 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:10.133+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:10.728 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:10.730+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:10.893 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:10.895+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:11.092 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:11.094+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:11.737 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:11.739+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:11.908 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:11.910+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:12.117 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:12.119+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:12.769 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:12.771+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:12.922 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:12.924+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:13.161 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:13.163+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:13.786 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:13.788+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:13.959 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:13.961+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:14.211 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:14.213+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:14.741 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:14.743+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:14.910 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:14.912+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:15.167 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:15.169+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:15.747 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:15.749+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:15.915 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:15.917+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:16.164 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:16.166+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:16.761 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:16.763+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:16.894 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:16.895+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:17.155 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:17.157+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:17.743 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:17.745+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:17.926 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:17.928+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:18.183 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:18.185+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:18.737 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:18.739+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:18.953 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:18.955+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:19.211 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:19.213+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:19.708 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:19.709+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:19.968 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:19.970+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:20.187 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:20.189+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:20.751 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:20.753+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:20.927 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:20.929+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:21.221 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:21.223+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:21.738 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:21.740+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:21.933 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:21.935+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:22.238 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:22.240+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:22.698 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:22.700+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:22.978 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:22.980+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:23.243 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:23.245+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:23.691 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:23.693+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:23.959 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:23.961+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:24.263 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:24.265+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:24.671 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:24.673+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:24.939 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:24.940+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:25.260 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:25.261+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:25.715 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:25.717+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:25.978 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:25.980+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:26.284 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:26.286+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:26.721 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:26.723+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:27.000 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:27.001+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:27.309 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:27.311+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:27.743 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:27.745+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:27.986 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:27.988+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:28.325 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:28.327+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:28.765 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:28.767+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:28.948 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:28.949+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:29.369 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:29.371+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:29.735 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:29.736+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:29.926 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:29.927+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:30.358 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:30.360+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:30.754 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:30.756+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:30.913 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:30.915+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:31.390 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:31.391+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:31.740 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:31.742+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:31.893 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:31.895+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:32.433 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:32.435+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:32.704 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:32.706+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:32.873 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:32.875+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:33.387 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:33.389+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:33.705 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:33.707+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:33.904 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:33.906+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:34.345 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:34.347+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:34.689 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:34.691+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:34.914 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:34.917+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:35.365 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:35.366+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:35.722 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:35.725+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:35.910 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:35.913+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:36.330 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:36.333+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:36.740 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:36.743+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:36.865 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:36.868+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:37.286 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:37.289+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:37.768 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:37.771+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:37.821 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:37.824+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:38.269 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:38.271+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:38.726 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:38.728+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:38.772 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:38.775+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:39.244 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:39.247+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:39.746 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:39.749+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:39.815 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:39.817+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:40.277 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:40.280+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:40.724 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:40.727+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:40.860 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:40.863+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:41.230 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:41.233+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:41.710 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:41.712+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:41.885 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:41.888+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:42.192 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:42.193+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:42.750 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:42.753+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:42.924 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:42.927+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:43.226 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:43.228+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:43.765 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:43.767+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:43.956 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:43.959+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:44.194 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:44.196+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:44.756 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:44.759+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:44.980 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:44.983+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:45.206 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:45.208+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:45.712 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:45.715+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:45.970 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:45.972+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:46.236 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:46.239+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:46.679 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:46.681+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:46.927 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:46.930+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:47.196 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:47.199+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:47.650 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:47.653+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:47.932 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:47.935+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:48.157 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:48.159+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:48.650 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:48.653+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:48.961 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:48.964+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:49.164 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:49.166+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:49.693 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:49.695+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:49.974 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:49.976+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:50.154 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:50.156+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:50.701 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:50.703+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:50.937 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:50.939+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:51.151 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:51.153+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:51.707 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:51.708+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:51.950 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:51.951+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:52.113 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:52.115+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:52.757 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:52.759+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:52.985 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:52.987+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:53.121 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:53.123+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:53.750 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:53.751+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:54.034 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:54.036+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:54.090 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:54.092+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:54.787 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:54.789+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:55.043 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:55.045+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:55.064 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:55.066+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:55.813 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:55.814+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:56.011 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:56.013+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:56.098 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:56.100+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:56.796 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:56.797+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:57.043 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:57.044+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:57.108 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:57.110+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:57.748 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:57.750+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:58.090 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:58.092+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:58.146 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:58.148+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:58.754 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:58.756+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:59.080 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:12:59.082+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:59.119 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:12:59.121+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:12:59.718 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:12:59.720+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:00.102 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:00.104+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:00.120 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:00.122+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:00.735 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:00.737+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:01.079 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:01.080+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:01.146 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:01.147+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:01.688 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:01.690+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:02.099 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:02.101+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:02.142 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:02.144+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:02.652 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:02.654+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:03.065 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:03.067+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:03.176 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:03.178+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:03.637 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:03.638+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:04.047 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:04.048+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:04.168 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:04.169+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:04.618 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:04.620+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:05.005 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:05.006+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:05.180 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:05.182+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:05.598 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:05.599+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:06.031 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:06.032+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:06.181 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:06.182+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:06.552 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:06.554+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:07.024 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:07.026+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:07.134 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:07.136+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:07.537 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:07.539+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:07.989 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:07.991+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:08.092 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:08.094+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:08.547 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:08.549+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:08.969 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:08.971+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:09.082 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:09.084+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:09.520 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:09.521+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:10.001 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:10.003+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:10.057 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:10.059+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:10.552 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:10.553+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:10.996 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:10.998+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:11.042 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:11.044+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:11.564 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:11.564+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:11.967 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:11.969+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:12.087 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:12.089+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:12.593 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:12.594+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:12.965 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:12.966+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:13.106 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:13.108+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:13.600 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:13.601+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:14.007 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:14.009+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:14.145 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:14.146+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:14.646 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:14.648+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:14.963 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:14.965+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:15.104 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:15.106+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:15.648 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:15.650+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:15.987 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:15.989+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:16.102 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:16.104+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:16.644 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:16.646+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:16.983 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:16.984+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:17.063 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:17.064+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:17.625 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:17.627+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:17.962 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:17.963+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:18.092 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:18.094+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:18.611 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:18.612+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:18.999 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:19.001+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:19.128 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:19.130+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:19.652 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:19.652+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:20.046 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:20.049+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:20.153 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:20.154+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:20.615 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:20.616+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:21.072 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:21.075+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:21.114 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:21.116+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:21.574 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:21.577+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:22.077 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:22.080+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:22.090 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:22.093+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:22.534 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:22.536+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:23.062 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:23.065+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:23.084 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:23.087+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:23.583 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:23.585+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:24.015 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:24.018+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:24.087 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:24.090+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:24.596 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:24.598+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:24.987 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:24.990+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:25.100 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:25.102+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:25.565 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:25.567+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:25.967 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:25.970+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:26.064 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:26.067+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:26.610 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:26.612+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:27.012 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:27.014+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:27.098 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:27.100+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:27.592 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:27.595+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:27.968 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:27.971+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:28.072 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:28.075+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:28.594 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:28.597+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:28.931 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:28.934+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:29.109 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:29.112+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:29.592 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:29.595+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:29.888 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:29.891+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:30.085 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:30.087+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:30.619 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:30.621+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:30.842 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:30.845+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:31.114 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:31.116+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:31.638 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:31.640+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:31.827 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:31.830+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:32.150 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:32.153+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:32.656 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:32.659+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:32.866 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:32.868+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:33.161 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:33.163+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:33.632 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:33.635+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:33.887 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:33.890+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:34.121 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:34.122+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:34.662 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:34.664+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:34.881 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:34.883+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:35.111 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:35.113+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:35.680 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:35.681+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:35.880 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:35.882+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:36.152 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:36.154+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:36.679 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:36.680+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:36.871 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:36.873+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:37.197 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:37.196+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:37.701 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:37.702+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:37.869 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:37.871+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:38.186 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:38.188+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:38.671 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:38.673+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:38.833 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:38.835+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:39.200 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:39.202+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:39.626 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:39.628+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:39.795 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:39.796+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:40.215 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:40.216+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:40.666 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:40.668+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:40.784 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:40.786+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:41.248 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:41.250+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:41.702 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:41.703+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:41.758 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:41.760+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:42.293 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:42.295+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:42.673 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:42.674+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:42.748 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:42.749+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:43.246 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:43.247+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:43.681 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:43.682+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:43.715 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:43.716+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:44.263 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:44.264+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:44.670 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:44.672+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:44.756 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:44.757+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:45.271 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:45.273+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:45.700 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:45.702+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:45.783 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:45.785+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:46.241 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:46.243+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:46.712 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:46.714+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:46.743 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:46.745+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:47.222 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:47.223+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:47.688 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:47.690+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:47.764 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:47.766+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:48.225 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:48.226+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:48.729 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:48.730+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:48.808 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:48.809+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:49.178 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:49.180+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:49.735 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:49.736+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:49.784 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:49.786+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:50.159 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:50.160+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:50.762 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:50.764+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:50.771 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:50.772+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:51.206 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:51.208+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:51.751 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:51.752+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:51.814 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:51.815+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:52.186 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:52.188+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:52.751 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:52.753+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:52.836 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:52.838+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:53.147 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:53.149+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:53.701 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:53.703+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:53.788 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:53.789+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:54.166 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:54.168+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:54.696 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:54.697+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:54.810 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:54.811+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:55.132 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:55.134+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:55.703 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:55.704+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:55.770 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:55.771+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:56.160 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:56.162+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:56.692 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:56.693+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:56.748 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:56.750+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:57.118 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:57.119+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:57.646 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:57.648+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:57.728 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:57.729+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:58.113 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:58.114+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:58.663 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:58.665+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:58.745 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:58.747+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:59.100 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:13:59.102+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:59.658 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:13:59.659+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:13:59.745 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:13:59.747+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:00.058 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:00.060+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:00.701 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:00.702+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:00.763 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:00.764+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:01.062 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:01.064+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:01.742 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:01.744+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:01.764 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:01.765+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:02.065 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:02.066+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:02.703 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:02.705+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:02.729 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:02.730+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:03.111 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:03.112+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:03.697 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:03.698+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:03.776 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:03.777+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:04.114 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:04.115+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:04.738 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:04.740+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:04.806 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:04.808+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:05.144 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:05.146+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:05.784 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:05.786+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:05.848 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:05.851+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:06.165 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:06.168+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:06.827 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:06.829+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:06.829 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:06.832+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:07.173 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:07.175+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:07.791 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:07.793+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:07.838 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:07.840+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:08.182 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:08.184+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:08.761 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:08.763+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:08.797 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:08.799+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:09.213 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:09.216+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:09.749 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:09.750+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:09.838 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:09.841+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:10.194 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:10.196+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:10.779 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:10.782+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:10.855 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:10.858+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:11.201 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:11.203+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:11.739 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:11.741+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:11.839 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:11.841+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:12.227 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:12.230+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:12.746 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:12.748+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:12.799 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:12.801+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:13.229 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:13.232+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:13.762 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:13.765+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:13.780 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:13.783+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:14.189 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:14.192+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:14.744 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:14.746+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:14.774 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:14.777+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:15.229 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:15.232+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:15.700 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:15.703+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:15.763 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:15.766+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:16.194 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:16.196+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:16.709 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:16.711+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:16.800 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:16.802+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:17.151 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:17.154+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:17.675 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:17.677+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:17.774 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:17.776+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:18.163 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:18.166+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:18.722 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:18.724+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:18.764 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:18.767+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:19.185 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:19.187+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:19.745 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:19.747+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:19.748 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:19.749+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:20.202 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:20.203+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:20.705 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:20.707+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:20.780 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:20.781+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:21.205 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:21.207+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:21.709 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:21.711+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:21.778 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:21.779+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:22.241 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:22.243+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:22.682 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:22.683+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:22.733 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:22.735+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:23.222 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:23.223+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:23.670 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:23.671+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:23.688 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:23.690+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:24.272 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:24.273+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:24.679 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:24.681+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:24.697 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:24.699+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:25.314 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:25.316+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:25.637 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:25.639+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:25.673 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:25.674+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:26.281 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:26.283+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:26.624 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:26.625+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:26.652 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:26.654+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:27.251 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:27.253+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:27.620 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:27.621+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:27.656 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:27.658+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:28.204 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:28.206+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:28.630 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:28.632+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:28.676 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:28.678+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:29.252 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:29.253+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:29.642 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:29.644+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:29.678 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:29.679+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:30.273 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:30.275+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:30.644 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:30.646+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:30.685 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:30.686+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:31.255 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:31.257+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:31.649 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:31.651+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:31.650 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:31.651+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:32.292 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:32.294+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:32.616 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:32.618+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:32.672 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:32.673+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:33.259 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:33.261+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:33.586 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:33.587+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:33.646 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:33.647+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:34.241 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:34.243+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:34.539 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:34.541+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:34.635 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:34.637+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:35.268 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:35.269+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:35.503 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:35.504+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:35.678 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:35.679+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:36.255 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:36.257+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:36.456 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:36.457+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:36.685 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:36.686+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:37.223 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:37.224+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:37.453 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:37.454+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:37.710 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:37.711+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:38.225 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:38.227+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:38.481 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:38.482+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:38.697 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:38.697+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:39.181 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:39.183+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:39.462 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:39.463+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:39.744 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:39.745+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:40.138 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:40.135+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:40.503 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:40.504+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:40.741 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:40.743+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:41.091 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:41.092+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:41.546 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:41.548+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:41.749 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:41.751+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:42.126 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:42.127+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:42.570 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:42.571+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:42.718 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:42.720+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:43.107 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:43.109+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:43.604 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:43.606+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:43.758 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:43.760+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:44.117 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:44.119+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:44.639 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:44.640+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:44.711 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:44.713+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:45.140 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:45.142+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:45.649 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:45.650+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:45.728 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:45.729+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:46.179 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:46.180+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:46.682 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:46.684+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:46.700 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:46.702+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:47.133 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:47.135+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:47.698 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:47.699+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:47.701 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:47.703+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:48.131 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:48.132+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:48.657 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:48.658+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:48.675 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:48.675+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:49.168 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:49.169+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:49.692 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:49.693+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:49.715 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:49.716+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:50.165 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:50.165+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:50.695 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:50.698+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:50.752 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:50.754+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:51.156 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:51.158+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:51.673 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:51.676+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:51.712 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:51.715+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:52.170 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:52.173+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:52.699 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:52.701+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:52.718 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:52.720+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:53.181 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:53.184+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:53.692 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:53.694+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:53.756 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:53.758+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:54.156 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:54.158+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:54.644 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:54.646+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:54.774 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:54.777+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:55.164 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:55.167+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:55.614 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:55.616+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:55.774 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:55.777+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:56.212 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:56.214+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:56.630 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:56.632+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:56.758 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:56.760+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:57.242 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:57.244+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:57.605 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:57.607+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:57.712 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:57.714+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:58.279 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:58.281+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:58.567 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:58.569+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:58.752 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:58.754+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:59.266 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:14:59.269+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:59.598 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:14:59.601+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:14:59.745 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:14:59.747+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:00.253 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:00.256+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:00.561 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:00.564+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:00.775 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:00.777+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:01.304 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:01.306+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:01.522 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:01.525+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:01.731 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:01.733+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:02.315 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:02.317+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:02.547 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:02.549+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:02.702 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:02.705+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:03.311 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:03.314+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:03.568 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:03.570+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:03.672 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:03.674+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:04.304 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:04.305+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:04.530 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:04.531+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:04.678 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:04.679+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:05.257 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:05.259+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:05.481 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:05.483+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:05.686 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:05.687+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:06.265 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:06.267+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:06.440 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:06.441+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:06.722 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:06.723+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:07.229 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:07.231+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:07.474 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:07.475+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:07.678 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:07.680+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:08.182 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:08.183+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:08.514 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:08.515+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:08.674 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:08.675+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:09.202 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:09.203+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:09.518 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:09.519+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:09.691 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:09.692+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:10.164 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:10.165+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:10.500 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:10.502+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:10.671 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:10.672+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:11.182 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:11.183+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:11.476 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:11.478+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:11.673 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:11.675+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:12.218 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:12.219+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:12.525 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:12.527+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:12.706 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:12.708+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:13.180 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:13.181+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:13.482 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:13.483+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:13.730 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:13.731+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:14.175 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:14.176+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:14.435 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:14.436+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:14.772 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:14.771+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:15.205 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:15.206+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:15.399 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:15.400+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:15.748 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:15.749+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:16.212 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:16.213+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:16.413 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:16.415+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:16.747 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:16.749+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:17.262 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:17.263+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:17.365 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:17.366+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:17.721 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:17.723+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:18.240 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:18.241+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:18.376 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:18.377+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:18.693 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:18.694+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:19.203 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:19.204+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:19.354 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:19.355+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:19.711 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:19.712+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:20.198 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:20.199+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:20.367 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:20.369+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:20.689 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:20.690+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:21.191 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:21.192+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:21.359 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:21.360+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:21.660 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:21.661+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:22.181 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:22.183+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:22.373 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:22.374+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:22.633 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:22.634+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:23.191 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:23.193+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:23.383 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:23.384+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:23.639 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:23.640+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:24.174 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:24.175+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:24.409 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:24.411+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:24.669 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:24.670+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:25.211 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:25.212+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:25.447 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:25.449+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:25.667 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:25.668+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:26.216 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:26.217+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:26.420 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:26.421+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:26.692 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:26.693+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:27.264 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:27.265+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:27.420 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:27.421+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:27.718 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:27.719+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:28.267 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:28.268+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:28.440 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:28.441+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:28.738 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:28.739+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:29.257 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:29.259+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:29.400 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:29.401+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:29.777 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:29.778+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:30.250 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:30.251+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:30.428 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:30.429+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:30.821 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:30.822+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:31.210 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:31.211+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:31.474 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:31.475+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:31.844 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:31.845+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:32.200 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:32.201+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:32.449 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:32.450+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:32.875 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:32.876+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:33.185 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:33.186+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:33.492 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:33.493+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:33.846 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:33.847+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:34.174 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:34.176+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:34.524 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:34.525+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:34.882 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:34.884+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:35.197 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:35.199+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:35.550 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:35.552+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:35.845 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:35.847+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:36.247 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:36.250+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:36.562 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:36.564+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:36.850 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:36.852+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:37.285 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:37.288+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:37.541 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:37.544+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:37.814 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:37.816+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:38.259 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:38.261+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:38.507 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:38.509+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:38.816 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:38.818+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:39.303 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:39.305+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:39.495 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:39.498+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:39.848 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:39.850+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:40.292 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:40.294+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:40.504 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:40.506+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:40.880 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:40.882+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:41.300 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:41.302+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:41.538 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:41.540+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:41.879 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:41.881+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:42.252 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:42.254+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:42.563 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:42.565+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:42.861 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:42.863+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:43.289 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:43.291+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:43.602 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:43.604+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:43.812 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:43.814+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:44.248 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:44.250+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:44.630 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:44.632+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:44.837 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:44.839+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:45.225 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:45.227+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:45.649 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:45.651+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:45.847 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:45.849+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:46.180 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:46.182+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:46.604 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:46.607+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:46.797 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:46.799+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:47.180 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:47.182+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:47.655 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:47.658+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:47.837 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:47.839+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:48.187 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:48.189+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:48.683 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:48.684+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:48.862 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:48.863+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:49.166 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:49.167+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:49.709 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:49.710+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:49.892 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:49.893+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:50.184 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:50.186+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:50.696 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:50.697+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:50.881 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:50.882+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:51.152 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:51.153+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:51.743 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:51.745+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:51.856 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:51.857+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:52.142 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:52.143+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:52.705 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:52.707+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:52.865 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:52.866+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:53.155 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:53.156+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:53.720 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:53.721+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:53.911 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:53.912+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:54.114 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:54.115+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:54.685 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:54.687+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:54.877 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:54.878+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:55.163 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:55.164+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:55.713 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:55.715+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:55.915 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:55.916+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:56.134 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:56.136+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:56.732 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:56.733+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:56.906 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:56.907+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:57.117 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:57.118+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:57.711 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:57.712+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:57.879 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:57.880+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:58.069 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:58.070+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:58.665 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:58.667+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:58.842 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:58.843+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:59.105 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:15:59.106+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:59.665 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:15:59.667+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:15:59.891 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:15:59.892+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:00.064 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:00.065+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:00.686 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:00.687+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:00.859 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:00.861+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:01.067 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:01.068+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:01.657 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:01.658+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:01.861 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:01.862+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:02.098 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:02.099+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:02.683 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:02.684+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:02.812 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:02.813+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:03.075 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:03.077+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:03.657 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:03.658+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:03.850 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:03.852+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:04.063 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:04.064+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:04.701 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:04.702+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:04.898 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:04.899+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:05.043 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:05.044+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:05.684 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:05.685+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:05.885 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:05.886+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:06.088 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:06.089+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:06.640 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:06.641+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:06.893 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:06.894+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:07.088 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:07.089+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:07.608 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:07.609+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:07.888 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:07.889+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:08.082 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:08.083+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:08.616 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:08.617+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:08.932 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:08.933+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:09.099 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:09.101+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:09.665 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:09.666+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:09.897 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:09.896+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:10.063 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:10.064+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:10.674 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:10.675+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:10.935 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:10.936+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:11.084 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:11.085+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:11.631 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:11.633+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:11.927 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:11.928+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:12.075 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:12.076+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:12.656 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:12.657+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:12.959 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:12.960+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:13.085 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:13.086+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:13.688 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:13.689+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:13.983 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:13.985+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:14.116 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:14.117+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:14.646 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:14.647+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:14.989 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:14.990+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:15.101 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:15.103+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:15.666 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:15.667+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:16.015 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:16.016+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:16.076 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:16.077+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:16.628 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:16.629+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:17.003 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:17.004+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:17.070 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:17.071+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:17.601 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:17.602+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:18.020 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:18.021+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:18.043 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:18.044+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:18.638 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:18.639+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:19.026 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:19.027+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:19.040 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:19.041+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:19.652 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:19.653+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:20.004 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:20.004+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:20.055 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:20.056+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:20.668 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:20.670+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:20.996 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:20.998+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:21.022 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:21.024+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:21.712 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:21.714+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:22.009 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:22.011+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:22.013 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:22.015+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:22.679 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:22.682+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:23.015 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:23.017+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:23.055 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:23.057+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:23.695 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:23.697+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:24.038 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:24.040+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:24.054 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:24.056+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:24.724 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:24.726+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:25.068 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:25.070+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:25.079 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:25.081+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:25.710 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:25.712+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:26.038 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:26.040+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:26.044 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:26.046+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:26.704 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:26.706+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:26.993 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:26.995+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:27.009 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:27.011+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:27.685 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:27.687+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:28.023 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:28.025+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:28.045 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:28.047+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:28.700 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:28.702+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:29.040 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:29.043+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:29.058 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:29.060+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:29.734 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:29.736+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:30.048 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:30.050+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:30.085 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:30.087+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:30.742 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:30.744+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:31.038 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:31.040+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:31.088 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:31.090+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:31.759 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:31.761+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:32.006 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:32.008+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:32.116 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:32.118+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:32.722 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:32.724+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:32.968 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:32.970+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:33.124 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:33.126+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:33.702 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:33.703+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:33.987 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:33.988+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:34.108 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:34.109+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:34.739 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:34.740+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:34.996 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:34.997+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:35.143 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:35.144+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:35.744 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:35.745+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:36.021 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:36.022+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:36.186 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:36.187+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:36.760 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:36.761+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:37.047 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:37.048+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:37.159 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:37.160+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:37.715 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:37.716+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:38.080 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:38.081+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:38.160 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:38.161+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:38.741 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:38.742+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:39.042 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:39.043+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:39.154 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:39.155+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:39.713 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:39.714+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:40.055 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:40.056+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:40.138 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:40.139+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:40.740 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:40.741+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:41.097 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:41.099+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:41.168 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:41.169+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:41.786 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:41.787+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:42.090 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:42.091+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:42.142 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:42.143+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:42.767 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:42.768+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:43.061 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:43.062+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:43.176 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:43.177+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:43.780 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:43.781+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:44.019 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:44.020+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:44.164 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:44.165+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:44.808 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:44.809+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:45.002 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:45.003+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:45.196 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:45.197+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:45.794 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:45.795+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:45.985 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:45.986+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:46.205 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:46.206+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:46.749 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:46.750+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:46.957 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:46.959+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:47.236 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:47.237+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:47.768 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:47.769+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:47.978 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:47.979+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:48.272 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:48.273+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:48.734 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:48.735+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:48.998 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:48.999+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:49.285 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:49.286+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:49.688 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:49.689+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:49.989 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:49.990+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:50.324 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:50.325+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:50.640 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:50.641+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:51.029 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:51.029+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:51.322 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:51.323+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:51.664 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:51.665+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:51.986 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:51.986+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:52.283 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:52.284+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:52.618 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:52.619+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:52.947 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:52.948+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:53.250 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:53.251+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:53.596 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:53.597+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:53.914 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:53.915+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:54.224 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:54.225+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:54.620 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:54.621+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:54.903 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:54.904+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:55.250 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:55.251+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:55.669 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:55.670+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:55.897 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:55.898+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:56.238 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:56.239+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:56.624 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:56.625+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:56.855 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:56.856+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:57.268 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:57.269+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:57.668 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:57.669+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:57.904 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:57.905+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:58.306 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:58.307+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:58.690 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:58.691+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:58.918 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:58.919+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:59.292 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:16:59.293+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:59.676 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:16:59.677+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:16:59.957 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:16:59.958+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:00.312 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:00.313+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:00.702 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:00.703+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:00.994 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:00.995+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:01.318 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:01.319+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:01.659 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:01.660+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:02.012 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:02.013+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:02.281 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:02.282+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:02.675 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:02.676+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:02.966 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:02.967+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:03.278 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:03.279+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:03.669 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:03.670+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:04.002 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:04.003+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:04.289 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:04.290+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:04.687 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:04.688+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:05.027 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:05.028+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:05.256 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:05.258+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:05.730 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:05.732+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:06.004 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:06.006+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:06.255 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:06.257+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:06.696 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:06.698+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:06.976 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:06.978+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:07.283 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:07.285+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:07.650 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:07.652+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:07.965 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:07.967+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:08.303 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:08.305+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:08.688 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:08.690+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:08.971 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:08.973+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:09.295 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:09.297+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:09.640 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:09.642+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:09.930 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:09.932+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:10.257 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:10.259+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:10.642 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:10.644+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:10.925 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:10.927+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:11.269 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:11.270+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:11.310 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:17:11.312+0000 7fccf4bd9700 0 -- 172.21.15.87:0/3193322219 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x564162b40680 msgr2=0x564162aa91e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-24T18:17:11.672 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:11.674+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:11.900 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:11.902+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:12.317 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:12.319+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:12.681 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:12.683+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:12.890 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:12.892+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:13.292 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:13.294+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:13.642 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:13.644+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:13.928 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:13.930+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:14.312 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:14.314+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:14.689 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:14.691+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:14.959 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:14.961+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:15.322 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:15.324+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:15.711 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:15.713+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:15.942 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:15.943+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:16.314 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:16.316+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:16.680 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:16.682+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:16.923 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:16.925+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:17.284 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:17.286+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:17.646 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:17.648+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:17.940 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:17.942+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:18.265 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:18.266+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:18.628 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:18.629+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:18.977 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:18.978+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:19.309 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:19.309+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:19.586 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:19.587+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:19.938 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:19.939+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:20.294 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:20.295+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:20.620 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:20.621+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:20.960 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:20.961+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:21.274 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:21.275+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:21.632 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:21.633+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:21.964 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:21.965+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:22.233 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:22.234+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:22.590 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:22.591+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:23.007 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:23.007+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:23.256 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:23.257+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:23.638 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:23.639+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:23.997 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:23.998+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:24.226 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:24.227+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:24.590 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:24.591+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:25.011 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:25.012+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:25.239 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:25.240+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:25.635 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:25.636+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:25.976 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:25.977+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:26.232 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:26.233+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:26.677 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:26.678+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:26.961 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:26.962+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:27.188 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:27.189+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:27.654 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:27.655+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:27.983 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:27.984+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:28.224 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:28.225+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:28.663 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:28.664+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:29.012 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:29.013+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:29.256 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:29.257+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:29.626 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:29.627+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:29.979 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:29.980+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:30.250 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:30.251+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:30.662 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:30.663+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:30.978 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:30.979+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:31.275 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:31.276+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:31.654 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:31.655+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:32.021 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:32.022+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:32.271 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:32.272+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:32.612 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:32.613+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:33.035 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:33.036+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:33.224 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:33.225+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:33.590 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:33.591+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:34.062 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:34.063+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:34.245 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:34.245+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:34.568 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:34.569+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:35.076 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:35.077+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:35.254 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:35.255+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:35.566 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:35.567+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:36.054 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:36.055+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:36.275 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:36.276+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:36.528 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:36.528+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:37.063 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:37.064+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:37.285 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:37.286+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:37.485 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:37.486+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:38.095 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:38.096+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:38.250 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:38.251+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:38.450 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:38.451+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:39.064 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:39.065+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:39.204 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:39.205+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:39.402 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:39.403+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:40.019 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:40.020+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:40.240 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:40.241+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:40.429 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:40.430+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:41.067 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:41.067+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:41.229 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:41.229+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:41.424 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:41.425+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:42.044 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:42.045+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:42.244 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:42.245+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:42.346 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:17:42.347+0000 7fccf4bd9700 0 -- 172.21.15.87:0/3193322219 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fccd8033f40 msgr2=0x7fccd803cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-24T18:17:42.408 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:42.409+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:43.002 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:43.003+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:43.268 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:43.269+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:43.431 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:43.432+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:44.019 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:44.019+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:44.260 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:44.261+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:44.450 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:44.451+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:44.986 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:44.987+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:45.306 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:45.307+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:45.441 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:45.442+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:45.982 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:45.982+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:46.261 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:46.262+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:46.467 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:46.468+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:47.014 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:47.015+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:47.253 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:47.253+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:47.452 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:47.453+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:47.982 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:47.983+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:48.301 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:48.302+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:48.465 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:48.465+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:49.021 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:49.022+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:49.307 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:49.308+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:49.466 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:49.466+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:50.062 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:50.064+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:50.331 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:50.333+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:50.468 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:50.470+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:51.061 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:51.062+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:51.292 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:51.294+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:51.451 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:51.453+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:52.053 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:52.054+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:52.291 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:52.293+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:52.444 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:52.446+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:53.063 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:53.065+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:53.255 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:53.257+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:53.486 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:53.488+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:54.072 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:54.074+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:54.253 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:54.255+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:54.447 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:54.448+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:55.053 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:55.054+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:55.234 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:55.236+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:55.479 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:55.481+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:56.059 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:56.061+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:56.202 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:56.203+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:56.499 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:56.501+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:57.064 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:57.066+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:57.220 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:57.222+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:57.462 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:57.464+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:58.112 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:58.114+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:58.213 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:58.215+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:58.443 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:58.444+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:59.138 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:17:59.140+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:59.184 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:17:59.186+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:17:59.452 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:17:59.454+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:00.110 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:00.112+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:00.184 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:00.186+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:00.468 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:00.470+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:01.142 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:01.144+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:01.204 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:01.206+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:01.491 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:01.492+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:02.150 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:02.152+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:02.212 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:02.214+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:02.488 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:02.490+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:03.181 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:03.181+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:03.200 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:03.201+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:03.456 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:03.457+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:04.152 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:04.153+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:04.215 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:04.216+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:04.494 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:04.494+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:05.174 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:05.175+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:05.240 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:05.241+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:05.517 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:05.517+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:06.176 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:06.177+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:06.267 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:06.268+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:06.556 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:06.557+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:07.145 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:07.146+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:07.254 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:07.255+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:07.539 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:07.539+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:08.131 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:08.132+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:08.273 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:08.273+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:08.562 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:08.563+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:09.113 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:09.113+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:09.281 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:09.282+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:09.600 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:09.601+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:10.115 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:10.116+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:10.269 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:10.270+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:10.605 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:10.603+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:11.135 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:11.136+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:11.245 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:11.246+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:11.585 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:11.586+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:12.177 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:12.178+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:12.216 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:12.217+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:12.610 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:12.611+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:13.132 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:13.133+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:13.241 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:13.242+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:13.608 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:13.608+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:14.177 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:14.178+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:14.228 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:14.229+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:14.603 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:14.604+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:15.211 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:15.212+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:15.249 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:15.250+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:15.618 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:15.619+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:16.234 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:16.235+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:16.246 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:16.247+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:16.661 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:16.662+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:17.254 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:17.254+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:17.281 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:17.282+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:17.655 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:17.656+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:18.274 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:18.275+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:18.288 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:18.289+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:18.680 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:18.681+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:19.261 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:19.261+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:19.269 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:19.270+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:19.714 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:19.714+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:20.291 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:20.291+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:20.294 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:20.295+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:20.738 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:20.739+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:21.341 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:21.341+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:21.341 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:21.341+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:21.692 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:21.693+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:22.293 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:22.294+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:22.387 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:22.388+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:22.656 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:22.657+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:23.250 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:23.251+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:23.393 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:23.393+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:23.636 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:23.637+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:24.280 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:24.280+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:24.373 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:24.374+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:24.658 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:24.659+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:25.280 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:25.281+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:25.419 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:25.419+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:25.662 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:25.663+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:26.273 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:26.274+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:26.463 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:26.463+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:26.683 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:26.684+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:27.322 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:27.323+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:27.499 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:27.500+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:27.635 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:27.636+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:28.320 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:28.321+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:28.461 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:28.462+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:28.612 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:28.613+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:29.340 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:29.341+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:29.428 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:29.429+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:29.645 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:29.646+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:30.307 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:30.308+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:30.413 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:30.414+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:30.644 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:30.645+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:31.354 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:31.355+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:31.450 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:31.450+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:31.648 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:31.649+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:32.351 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:32.352+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:32.403 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:32.404+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:32.657 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:32.659+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:33.341 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:33.342+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:33.413 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:33.414+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:33.675 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:33.676+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:34.375 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:34.376+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:34.381 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:34.382+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:34.632 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:34.633+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:35.353 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:35.355+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:35.418 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:35.420+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:35.624 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:35.625+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:36.402 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:36.403+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:36.453 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:36.455+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:36.666 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:36.668+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:37.439 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:37.440+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:37.471 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:37.473+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:37.666 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:37.668+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:38.446 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:38.447+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:38.477 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:38.479+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:38.645 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:38.647+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:39.427 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:39.429+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:39.451 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:39.452+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:39.656 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:39.658+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:40.411 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:40.413+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:40.431 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:40.432+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:40.622 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:40.624+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:41.450 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:41.452+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:41.459 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:41.461+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:41.632 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:41.634+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:42.435 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:42.437+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:42.500 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:42.502+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:42.657 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:42.658+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:43.418 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:43.420+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:43.525 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:43.526+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:43.693 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:43.695+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:44.411 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:44.413+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:44.478 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:44.480+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:44.732 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:44.733+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:45.447 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:45.449+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:45.498 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:45.500+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:45.769 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:45.771+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:46.415 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:46.417+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:46.520 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:46.522+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:46.796 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:46.797+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:47.380 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:47.382+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:47.503 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:47.505+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:47.770 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:47.772+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:48.413 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:48.414+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:48.491 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:48.492+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:48.780 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:48.780+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:49.375 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:49.376+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:49.505 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:49.506+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:49.805 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:49.806+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:50.415 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:50.415+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:50.545 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:50.546+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:50.776 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:50.777+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:51.369 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:51.369+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:51.508 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:51.509+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:51.792 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:51.793+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:52.320 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:52.321+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:52.538 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:52.539+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:52.764 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:52.765+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:53.331 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:53.332+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:53.539 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:53.539+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:53.800 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:53.801+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:54.376 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:54.377+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:54.563 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:54.564+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:54.796 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:54.796+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:55.359 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:55.360+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:55.540 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:55.541+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:55.803 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:55.804+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:56.351 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:56.352+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:56.508 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:56.509+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:56.827 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:56.828+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:57.379 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:57.380+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:57.484 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:57.485+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:57.809 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:57.809+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:58.409 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:58.409+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:58.503 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:58.503+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:58.842 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:58.842+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:59.392 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:18:59.393+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:59.512 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:18:59.513+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:18:59.827 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:18:59.828+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:00.420 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:00.421+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:00.560 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:00.561+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:00.862 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:00.863+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:01.465 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:01.466+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:01.565 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:01.565+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:01.840 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:01.841+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:02.462 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:02.463+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:02.594 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:02.594+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:02.855 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:02.856+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:03.484 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:03.485+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:03.640 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:03.641+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:03.812 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:03.813+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:04.448 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:04.449+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:04.656 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:04.657+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:04.782 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:04.782+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:05.406 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:05.406+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:05.654 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:05.655+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:05.764 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:05.765+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:06.399 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:06.399+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:06.646 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:06.647+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:06.754 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:06.755+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:07.379 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:07.380+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:07.612 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:07.612+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:07.753 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:07.753+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:08.394 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:08.395+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:08.617 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:08.618+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:08.742 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:08.743+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:09.428 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:09.429+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:09.612 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:09.613+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:09.726 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:09.727+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:10.475 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:10.475+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:10.572 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:10.573+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:10.685 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:10.686+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:11.498 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:11.498+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:11.590 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:11.591+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:11.676 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:11.677+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:12.503 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:12.504+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:12.549 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:12.550+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:12.698 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:12.699+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:13.481 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:13.481+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:13.546 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:13.547+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:13.674 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:13.675+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:14.482 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:14.482+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:14.538 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:14.539+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:14.696 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:14.697+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:15.449 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:15.450+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:15.552 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:15.553+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:15.724 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:15.725+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:16.489 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:16.489+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:16.579 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:16.580+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:16.740 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:16.741+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:17.484 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:17.485+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:17.578 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:17.579+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:17.741 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:17.742+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:18.511 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:18.512+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:18.598 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:18.599+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:18.747 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:18.747+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:19.554 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:19.556+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:19.600 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:19.601+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:19.726 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:19.726+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:20.588 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:20.590+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:20.623 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:20.625+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:20.764 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:20.765+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:21.539 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:21.541+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:21.646 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:21.648+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:21.744 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:21.746+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:22.506 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:22.508+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:22.636 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:22.637+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:22.740 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:22.741+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:23.523 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:23.525+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:23.639 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:23.641+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:23.731 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:23.732+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:24.564 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:24.566+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:24.663 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:24.665+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:24.687 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:24.688+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:25.552 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:25.554+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:25.663 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:25.664+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:25.688 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:25.689+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:26.557 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:26.559+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:26.637 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:26.639+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:26.655 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:26.656+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:27.567 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:27.568+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:27.619 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:27.621+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:27.669 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:27.671+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:28.578 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:28.580+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:28.598 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:28.599+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:28.644 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:28.646+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:29.603 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:29.605+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:29.625 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:29.627+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:29.651 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:29.653+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:30.643 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:30.644+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:30.645 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:30.647+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:30.671 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:30.672+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:31.652 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:31.654+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:31.665 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:31.667+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:31.688 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:31.690+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:32.622 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:32.623+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:32.625 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:32.627+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:32.654 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:32.656+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:33.608 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:33.609+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:33.610 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:33.611+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:33.693 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:33.693+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:34.578 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:34.579+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:34.643 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:34.644+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:34.649 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:34.649+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:35.612 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:35.612+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:35.629 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:35.630+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:35.684 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:35.685+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:36.613 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:36.613+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:36.633 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:36.634+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:36.703 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:36.704+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:37.618 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:37.619+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:37.641 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:37.642+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:37.752 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:37.753+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:38.635 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:38.636+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:38.687 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:38.688+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:38.772 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:38.772+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:39.620 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:39.621+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:39.649 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:39.650+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:39.805 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:39.806+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:40.632 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:40.632+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:40.663 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:40.663+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:40.786 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:40.786+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:41.640 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:41.641+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:41.651 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:41.652+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:41.755 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:41.755+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:42.675 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:42.676+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:42.683 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:42.684+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:42.745 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:42.746+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:43.628 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:43.629+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:43.701 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:43.702+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:43.781 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:43.781+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:44.614 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:44.614+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:44.658 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:44.659+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:44.798 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:44.799+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:45.597 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:45.597+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:45.677 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:45.678+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:45.789 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:45.789+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:46.578 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:46.578+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:46.645 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:46.645+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:46.782 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:46.782+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:47.529 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:47.530+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:47.625 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:47.626+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:47.781 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:47.782+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:48.484 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:48.485+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:48.627 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:48.628+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:48.752 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:48.753+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:49.448 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:49.449+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:49.630 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:49.631+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:49.752 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:49.753+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:50.455 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:50.454+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:50.590 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:50.591+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:50.713 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:50.714+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:51.457 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:51.457+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:51.541 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:51.542+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:51.703 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:51.704+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:52.476 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:52.477+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:52.511 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:52.511+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:52.711 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:52.712+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:53.479 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:53.479+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:53.525 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:53.526+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:53.747 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:53.748+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:54.485 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:54.486+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:54.527 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:54.528+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:54.729 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:54.730+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:55.458 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:55.458+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:55.501 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:55.502+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:55.765 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:55.766+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:56.482 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:56.482+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:56.518 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:56.518+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:56.758 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:56.759+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:57.464 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:57.465+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:57.515 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:57.516+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:57.796 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:57.796+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:58.487 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:58.488+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:58.500 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:58.501+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:58.798 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:58.798+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:59.491 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:19:59.492+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:59.517 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:19:59.518+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:19:59.765 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:19:59.766+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:00.528 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:00.529+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:00.551 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:00.551+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:00.731 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:00.731+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:01.517 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:01.518+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:01.529 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:01.530+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:01.694 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:01.694+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:02.555 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:02.556+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:02.566 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:02.568+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:02.680 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:02.680+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:03.564 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:03.565+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:03.567 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:03.569+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:03.724 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:03.724+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:04.556 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:04.557+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:04.596 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:04.597+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:04.761 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:04.761+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:05.569 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:05.570+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:05.600 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:05.602+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:05.805 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:05.807+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:06.538 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:06.540+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:06.620 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:06.621+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:06.778 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:06.780+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:07.514 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:07.515+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:07.610 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:07.611+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:07.756 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:07.758+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:08.498 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:08.500+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:08.603 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:08.605+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:08.735 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:08.737+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:09.529 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:09.531+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:09.556 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:09.557+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:09.711 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:09.713+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:10.514 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:10.516+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:10.603 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:10.604+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:10.726 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:10.727+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:11.493 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:11.495+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:11.614 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:11.600+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:11.761 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:11.762+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:12.460 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:12.462+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:12.595 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:12.597+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:12.804 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:12.805+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:13.455 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:13.457+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:13.574 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:13.576+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:13.787 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:13.789+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:14.484 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:14.486+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:14.558 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:14.560+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:14.824 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:14.825+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:15.486 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:15.488+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:15.554 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:15.556+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:15.790 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:15.792+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:16.496 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:16.498+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:16.522 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:16.524+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:16.748 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:16.750+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:17.495 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:17.497+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:17.524 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:17.525+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:17.798 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:17.800+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:18.514 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:18.515+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:18.516 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:18.517+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:18.781 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:18.781+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:19.534 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:19.534+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:19.554 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:19.555+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:19.830 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:19.830+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:20.534 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:20.535+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:20.571 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:20.572+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:20.857 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:20.858+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:21.564 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:21.565+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:21.578 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:21.579+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:21.876 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:21.877+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:22.566 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:22.566+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:22.574 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:22.575+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:22.829 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:22.829+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:23.538 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:23.538+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:23.589 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:23.590+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:23.784 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:23.784+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:24.506 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:24.506+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:24.584 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:24.585+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:24.774 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:24.774+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:25.505 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:25.505+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:25.600 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:25.601+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:25.746 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:25.747+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:26.490 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:26.491+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:26.590 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:26.590+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:26.725 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:26.725+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:27.494 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:27.494+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:27.614 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:27.614+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:27.685 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:27.686+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:28.452 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:28.452+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:28.604 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:28.604+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:28.642 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:28.643+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:29.485 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:29.485+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:29.641 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:29.642+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:29.648 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:29.649+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:30.488 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:30.489+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:30.619 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:30.619+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:30.626 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:30.627+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:31.503 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:31.504+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:31.596 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:31.597+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:31.636 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:31.636+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:32.474 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:32.475+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:32.604 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:32.605+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:32.673 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:32.674+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:33.516 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:33.517+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:33.633 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:33.634+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:33.721 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:33.721+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:34.525 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:34.525+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:34.637 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:34.638+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:34.693 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:34.694+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:35.485 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:35.485+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:35.663 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:35.663+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:35.685 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:35.686+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:36.455 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:36.455+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:36.617 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:36.617+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:36.689 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:36.689+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:37.487 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:37.488+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:37.596 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:37.597+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:37.737 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:37.737+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:38.520 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:38.521+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:38.600 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:38.601+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:38.701 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:38.701+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:39.564 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:39.565+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:39.614 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:39.615+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:39.725 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:39.726+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:40.579 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:40.579+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:40.589 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:40.589+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:40.752 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:40.752+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:41.561 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:41.561+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:41.617 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:41.617+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:41.734 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:41.734+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:42.574 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:42.574+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:42.605 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:42.606+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:42.722 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:42.722+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:43.620 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:43.620+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:43.640 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:43.641+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:43.719 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:43.719+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:44.618 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:44.619+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:44.678 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:44.678+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:44.727 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:44.728+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:45.622 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:45.623+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:45.661 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:45.661+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:45.707 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:45.707+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:46.580 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:46.580+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:46.617 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:46.617+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:46.724 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:46.725+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:47.560 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:47.561+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:47.650 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:47.651+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:47.769 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:47.770+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:48.516 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:48.517+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:48.611 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:48.611+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:48.744 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:48.746+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:49.518 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:49.519+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:49.617 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:49.617+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:49.776 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:49.777+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:50.535 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:50.537+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:50.605 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:50.606+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:50.818 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:50.820+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:51.505 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:51.506+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:51.655 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:51.657+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:51.829 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:51.831+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:52.459 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:52.461+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:52.688 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:52.689+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:52.812 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:52.814+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:53.461 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:53.463+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:53.733 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:53.735+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:53.843 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:53.845+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:54.493 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:54.495+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:54.773 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:54.775+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:54.869 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:54.870+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:55.484 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:55.486+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:55.756 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:55.757+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:55.841 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:55.843+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:56.519 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:56.521+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:56.711 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:56.713+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:56.827 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:56.829+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:57.508 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:57.510+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:57.703 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:57.705+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:57.876 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:57.878+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:58.504 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:58.506+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:58.687 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:58.689+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:58.867 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:58.868+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:59.519 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:20:59.520+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:59.659 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:20:59.661+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:20:59.892 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:20:59.893+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:00.513 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:00.515+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:00.679 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:00.680+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:00.941 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:00.942+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:01.477 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:01.478+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:01.700 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:01.702+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:01.967 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:01.968+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:02.449 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:02.450+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:02.695 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:02.695+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:02.957 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:02.958+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:03.499 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:03.500+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:03.742 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:03.742+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:03.975 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:03.976+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:04.496 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:04.496+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:04.757 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:04.758+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:05.003 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:05.004+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:05.505 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:05.506+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:05.719 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:05.720+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:06.038 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:06.038+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:06.552 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:06.552+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:06.736 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:06.736+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:07.066 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:07.066+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:07.504 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:07.504+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:07.732 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:07.732+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:08.054 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:08.054+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:08.518 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:08.517+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:08.701 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:08.702+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:09.099 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:09.099+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:09.550 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:09.550+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:09.715 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:09.716+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:10.058 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:10.059+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:10.565 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:10.566+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:10.690 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:10.690+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:11.051 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:11.052+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:11.549 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:11.550+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:11.678 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:11.679+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:12.090 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:12.091+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:12.582 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:12.582+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:12.694 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:12.694+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:13.088 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:13.089+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:13.567 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:13.568+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:13.701 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:13.702+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:14.060 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:14.060+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:14.523 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:14.524+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:14.723 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:14.724+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:15.094 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:15.094+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:15.569 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:15.570+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:15.696 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:15.696+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:16.130 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:16.130+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:16.613 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:16.614+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:16.733 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:16.734+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:17.112 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:17.112+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:17.579 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:17.579+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:17.762 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:17.763+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:18.137 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:18.137+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:18.625 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:18.625+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:18.743 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:18.743+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:19.161 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:19.159+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:19.596 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:19.596+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:19.704 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:19.704+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:20.164 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:20.165+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:20.624 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:20.625+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:20.735 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:20.736+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:21.214 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:21.214+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:21.649 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:21.650+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:21.742 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:21.742+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:22.253 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:22.253+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:22.661 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:22.661+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:22.695 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:22.696+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:23.273 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:23.274+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:23.661 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:23.662+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:23.689 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:23.689+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:24.242 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:24.243+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:24.681 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:24.682+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:24.692 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:24.692+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:25.210 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:25.210+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:25.693 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:25.693+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:25.722 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:25.722+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:26.177 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:26.177+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:26.715 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:26.716+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:26.717 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:26.717+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:27.198 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:27.198+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:27.730 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:27.730+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:27.749 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:27.750+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:28.167 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:28.168+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:28.756 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:28.757+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:28.773 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:28.774+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:29.153 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:29.154+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:29.771 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:29.771+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:29.779 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:29.780+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:30.123 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:30.124+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:30.785 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:30.785+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:30.825 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:30.825+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:31.115 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:31.115+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:31.806 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:31.806+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:31.815 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:31.816+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:32.112 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:32.112+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:32.770 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:32.770+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:32.790 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:32.790+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:33.118 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:33.118+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:33.771 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:33.771+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:33.794 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:33.796+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:34.125 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:34.127+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:34.774 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:34.775+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:34.827 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:34.828+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:35.081 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:35.083+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:35.798 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:35.799+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:35.805 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:35.806+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:36.087 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:36.089+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:36.822 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:36.824+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:36.834 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:36.836+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:37.132 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:37.133+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:37.786 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:37.788+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:37.821 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:37.823+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:38.087 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:38.089+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:38.786 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:38.788+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:38.800 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:38.802+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:39.071 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:39.073+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:39.761 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:39.763+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:39.795 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:39.796+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:40.101 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:40.103+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:40.751 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:40.752+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:40.822 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:40.824+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:41.114 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:41.115+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:41.791 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:41.792+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:41.865 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:41.867+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:42.096 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:42.098+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:42.807 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:42.808+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:42.852 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:42.854+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:43.068 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:43.070+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:43.808 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:43.808+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:43.827 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:43.829+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:44.106 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:44.107+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:44.764 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:44.765+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:44.865 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:44.867+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:45.087 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:45.089+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:45.719 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:45.720+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:45.855 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:45.856+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:46.121 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:46.123+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:46.696 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:46.698+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:46.812 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:46.813+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:47.090 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:47.091+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:47.732 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:47.734+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:47.817 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:47.818+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:48.109 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:48.109+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:48.723 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:48.723+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:48.777 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:48.777+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:49.100 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:49.101+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:49.760 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:49.761+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:49.808 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:49.808+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:50.134 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:50.135+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:50.726 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:50.727+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:50.774 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:50.774+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:51.100 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:51.100+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:51.729 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:51.730+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:51.761 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:51.761+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:52.059 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:52.060+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:52.717 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:52.718+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:52.764 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:52.765+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:53.071 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:53.072+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:53.721 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:53.721+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:53.726 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:53.727+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:54.053 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:54.054+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:54.748 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:54.748+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:54.753 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:54.754+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:55.046 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:55.046+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:55.708 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:55.708+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:55.710 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:55.711+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:56.069 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:56.069+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:56.674 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:56.675+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:56.714 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:56.715+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:57.023 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:57.023+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:57.701 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:57.701+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:57.758 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:57.758+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:58.067 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:58.068+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:58.734 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:58.734+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:58.777 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:58.777+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:59.099 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:21:59.100+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:59.775 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:21:59.775+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:21:59.807 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:21:59.807+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:00.143 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:00.143+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:00.770 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:00.771+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:00.836 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:00.837+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:01.104 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:01.105+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:01.733 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:01.733+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:01.875 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:01.875+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:02.129 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:02.129+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:02.713 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:02.714+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:02.862 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:02.862+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:03.165 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:03.165+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:03.665 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:03.665+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:03.839 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:03.839+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:04.197 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:04.198+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:04.686 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:04.687+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:04.814 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:04.814+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:05.193 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:05.194+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:05.728 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:05.728+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:05.853 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:05.853+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:06.182 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:06.183+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:06.707 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:06.707+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:06.878 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:06.879+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:07.228 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:07.228+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:07.719 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:07.719+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:07.872 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:07.873+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:08.269 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:08.270+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:08.769 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:08.770+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:08.847 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:08.848+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:09.272 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:09.273+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:09.792 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:09.792+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:09.825 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:09.823+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:10.283 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:10.284+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:10.809 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:10.810+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:10.852 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:10.853+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:11.260 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:11.261+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:11.763 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:11.764+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:11.872 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:11.872+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:12.258 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:12.259+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:12.784 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:12.785+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:12.847 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:12.847+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:13.251 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:13.252+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:13.763 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:13.763+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:13.854 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:13.855+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:14.210 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:14.210+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:14.749 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:14.750+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:14.828 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:14.829+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:15.238 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:15.239+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:15.705 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:15.705+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:15.782 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:15.783+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:16.278 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:16.278+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:16.745 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:16.745+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:16.752 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:16.752+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:17.324 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:17.324+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:17.707 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:17.708+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:17.752 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:17.753+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:18.356 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:18.356+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:18.711 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:18.711+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:18.752 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:18.753+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:19.319 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:19.320+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:19.694 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:19.696+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:19.775 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:19.777+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:20.334 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:20.335+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:20.666 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:20.667+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:20.788 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:20.789+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:21.365 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:21.367+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:21.666 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:21.667+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:21.750 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:21.751+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:22.413 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:22.414+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:22.623 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:22.624+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:22.778 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:22.779+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:23.462 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:23.463+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:23.609 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:23.611+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:23.758 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:23.759+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:24.501 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:24.502+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:24.591 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:24.593+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:24.755 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:24.757+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:25.551 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:25.552+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:25.583 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:25.585+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:25.762 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:25.764+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:26.576 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:26.577+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:26.625 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:26.627+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:26.787 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:26.788+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:27.537 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:27.539+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:27.655 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:27.657+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:27.824 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:27.825+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:28.549 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:28.550+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:28.687 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:28.688+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:28.838 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:28.840+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:29.577 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:29.578+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:29.731 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:29.733+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:29.812 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:29.814+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:30.528 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:30.529+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:30.778 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:30.779+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:30.822 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:30.823+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:31.507 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:31.509+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:31.733 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:31.733+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:31.851 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:31.852+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:32.522 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:32.524+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:32.734 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:32.735+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:32.822 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:32.822+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:33.499 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:33.499+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:33.728 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:33.728+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:33.827 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:33.827+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:34.515 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:34.515+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:34.710 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:34.710+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:34.867 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:34.868+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:35.508 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:35.508+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:35.742 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:35.742+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:35.836 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:35.836+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:36.528 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:36.529+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:36.742 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:36.742+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:36.801 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:36.802+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:37.563 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:37.563+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:37.751 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:37.752+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:37.830 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:37.830+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:38.578 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:38.579+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:38.761 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:38.762+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:38.878 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:38.878+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:39.562 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:39.562+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:39.718 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:39.718+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:39.871 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:39.871+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:40.527 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:40.527+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:40.743 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:40.743+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:40.880 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:40.880+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:41.498 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:41.498+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:41.745 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:41.746+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:41.836 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:41.837+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:42.448 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:42.449+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:42.789 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:42.790+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:42.810 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:42.810+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:43.488 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:43.488+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:43.784 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:43.784+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:43.790 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:43.790+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:44.501 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:44.501+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:44.737 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:44.737+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:44.774 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:44.775+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:45.465 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:45.465+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:45.725 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:45.725+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:45.749 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:45.750+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:46.443 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:46.443+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:46.712 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:46.712+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:46.795 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:46.795+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:47.408 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:47.409+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:47.679 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:47.679+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:47.831 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:47.831+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:48.452 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:48.453+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:48.716 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:48.716+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:48.861 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:48.862+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:49.435 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:49.436+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:49.731 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:49.732+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:49.816 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:49.816+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:50.449 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:50.449+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:50.705 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:50.705+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:50.800 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:50.800+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:51.451 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:51.451+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:51.691 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:51.692+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:51.779 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:51.780+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:52.461 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:52.461+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:52.671 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:52.671+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:52.753 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:52.754+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:53.435 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:53.435+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:53.703 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:53.703+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:53.728 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:53.728+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:54.480 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:54.480+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:54.665 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:54.666+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:54.734 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:54.735+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:55.432 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:55.433+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:55.708 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:55.708+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:55.741 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:55.741+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:56.410 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:56.410+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:56.691 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:56.692+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:56.759 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:56.759+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:57.438 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:57.439+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:57.733 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:57.734+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:57.783 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:57.783+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:58.448 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:58.448+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:58.743 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:58.744+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:58.747 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:58.748+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:59.456 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:22:59.457+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:59.750 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:22:59.750+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:22:59.763 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:22:59.764+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:00.421 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:00.421+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:00.782 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:00.782+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:00.800 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:00.800+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:01.431 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:01.431+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:01.743 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:01.743+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:01.787 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:01.788+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:02.456 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:02.458+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:02.766 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:02.767+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:02.785 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:02.786+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:03.462 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:03.463+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:03.770 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:03.771+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:03.813 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:03.814+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:04.462 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:04.463+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:04.770 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:04.771+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:04.862 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:04.863+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:05.478 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:05.480+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:05.814 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:05.815+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:05.896 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:05.898+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:06.517 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:06.518+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:06.776 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:06.777+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:06.921 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:06.922+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:07.541 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:07.542+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:07.813 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:07.814+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:07.967 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:07.968+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:08.580 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:08.581+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:08.861 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:08.860+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:08.942 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:08.944+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:09.598 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:09.599+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:09.862 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:09.864+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:09.910 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:09.911+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:10.615 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:10.616+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:10.863 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:10.865+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:10.901 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:10.903+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:11.585 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:11.587+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:11.858 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:11.859+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:11.910 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:11.911+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:12.605 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:12.606+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:12.904 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:12.906+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:12.939 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:12.941+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:13.616 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:13.617+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:13.894 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:13.896+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:13.962 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:13.964+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:14.654 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:14.655+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:14.879 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:14.880+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:14.955 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:14.956+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:15.671 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:15.672+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:15.926 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:15.928+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:15.994 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:15.996+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:16.650 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:16.651+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:16.946 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:16.947+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:16.970 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:16.972+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:17.610 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:17.612+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:17.933 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:17.933+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:18.006 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:18.006+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:18.573 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:18.574+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:18.887 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:18.887+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:18.980 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:18.980+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:19.568 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:19.568+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:19.873 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:19.874+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:19.974 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:19.975+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:20.600 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:20.600+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:20.856 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:20.856+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:20.982 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:20.982+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:21.597 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:21.597+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:21.892 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:21.892+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:22.024 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:22.024+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:22.567 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:22.568+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:22.877 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:22.877+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:23.069 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:23.069+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:23.528 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:23.528+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:23.862 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:23.863+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:24.110 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:24.110+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:24.577 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:24.577+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:24.912 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:24.912+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:25.062 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:25.062+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:25.614 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:25.614+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:25.948 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:25.949+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:26.015 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:26.015+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:26.581 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:26.582+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:26.950 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:26.950+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:27.012 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:27.012+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:27.593 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:27.593+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:27.908 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:27.909+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:27.990 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:27.990+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:28.576 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:28.576+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:28.953 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:28.954+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:29.030 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:29.030+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:29.598 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:29.598+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:29.990 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:29.990+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:30.074 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:30.075+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:30.612 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:30.612+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:30.962 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:30.963+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:31.028 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:31.029+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:31.654 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:31.655+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:31.940 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:31.940+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:32.007 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:32.007+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:32.625 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:32.625+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:32.954 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:32.955+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:33.024 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:33.024+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:33.649 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:33.649+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:33.930 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:33.930+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:34.056 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:34.056+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:34.642 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:34.643+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:34.937 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:34.937+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:35.020 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:35.020+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:35.628 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:35.628+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:35.909 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:35.909+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:36.001 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:36.001+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:36.650 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:36.651+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:36.936 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:36.937+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:37.039 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:37.039+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:37.611 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:37.611+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:37.936 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:37.937+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:38.050 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:38.051+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:38.575 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:38.576+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:38.961 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:38.961+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:39.082 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:39.083+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:39.602 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:39.603+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:39.961 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:39.961+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:40.109 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:40.109+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:40.624 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:40.624+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:40.956 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:40.957+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:41.061 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:41.061+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:41.624 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:41.625+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:41.944 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:41.945+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:42.079 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:42.079+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:42.626 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:42.626+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:42.899 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:42.900+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:43.058 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:43.059+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:43.580 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:43.580+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:43.855 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:43.855+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:44.073 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:44.074+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:44.587 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:44.588+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:44.857 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:44.857+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:45.118 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:45.119+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:45.571 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:45.572+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:45.880 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:45.880+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:46.145 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:46.145+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:46.522 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:46.522+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:46.874 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:46.875+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:47.139 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:47.139+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:47.538 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:47.538+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:47.921 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:47.921+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:48.104 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:48.105+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:48.510 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:48.511+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:48.878 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:48.879+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:49.065 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:49.065+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:49.498 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:49.498+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:49.918 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:49.919+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:50.083 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:50.085+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:50.485 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:50.487+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:50.870 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:50.872+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:51.080 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:51.081+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:51.479 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:51.480+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:51.836 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:51.837+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:52.060 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:52.058+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:52.459 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:52.460+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:52.842 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:52.843+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:53.107 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:53.108+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:53.488 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:53.489+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:53.796 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:53.798+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:54.082 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:54.084+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:54.486 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:54.488+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:54.844 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:54.846+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:55.053 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:55.054+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:55.509 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:55.510+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:55.818 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:55.819+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:56.005 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:56.007+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:56.532 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:56.533+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:56.792 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:56.793+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:56.970 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:56.971+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:57.521 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:57.522+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:57.826 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:57.827+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:57.985 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:57.986+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:58.476 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:58.477+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:58.827 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:58.829+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:58.991 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:23:58.992+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:59.476 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:23:59.478+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:23:59.796 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:23:59.797+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:00.036 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:00.037+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:00.509 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:00.510+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:00.810 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:00.812+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:01.013 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:01.014+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:01.541 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:01.542+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:01.858 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:01.860+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:01.998 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:02.000+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:02.511 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:02.512+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:02.855 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:02.855+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:03.034 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:03.034+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:03.520 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:03.520+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:03.906 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:03.906+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:04.067 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:04.068+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:04.548 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:04.549+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:04.908 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:04.908+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:05.043 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:05.043+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:05.591 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:05.591+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:05.897 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:05.897+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:06.005 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:06.005+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:06.588 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:06.588+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:06.875 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:06.875+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:06.998 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:06.998+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:07.566 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:07.566+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:07.853 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:07.853+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:07.951 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:07.951+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:08.571 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:08.572+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:08.845 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:08.845+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:08.928 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:08.925+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:09.537 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:09.538+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:09.888 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:09.888+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:09.963 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:09.963+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:10.517 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:10.517+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:10.854 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:10.854+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:10.935 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:10.935+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:11.541 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:11.541+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:11.870 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:11.870+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:11.937 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:11.937+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:12.550 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:12.551+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:12.845 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:12.846+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:12.914 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:12.914+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:13.574 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:13.574+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:13.824 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:13.824+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:13.908 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:13.908+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:14.548 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:14.548+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:14.804 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:14.805+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:14.867 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:14.867+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:15.563 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:15.563+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:15.757 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:15.757+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:15.849 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:15.849+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:16.544 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:16.544+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:16.708 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:16.708+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:16.832 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:16.833+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:17.513 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:17.513+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:17.745 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:17.746+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:17.860 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:17.861+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:18.489 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:18.489+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:18.701 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:18.701+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:18.822 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:18.822+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:19.495 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:19.495+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:19.691 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:19.691+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:19.840 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:19.840+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:20.480 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:20.480+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:20.713 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:20.713+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:20.797 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:20.796+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:21.504 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:21.504+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:21.754 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:21.754+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:21.815 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:21.815+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:22.490 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:22.490+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:22.755 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:22.755+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:22.859 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:22.860+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:23.498 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:23.498+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:23.758 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:23.759+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:23.894 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:23.895+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:24.463 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:24.464+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:24.757 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:24.757+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:24.928 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:24.929+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:25.459 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:25.459+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:25.801 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:25.801+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:25.945 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:25.945+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:26.479 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:26.479+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:26.800 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:26.800+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:26.933 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:26.933+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:27.503 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:27.503+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:27.826 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:27.827+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:27.937 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:27.937+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:28.462 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:28.462+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:28.788 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:28.788+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:28.948 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:28.948+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:29.432 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:29.433+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:29.804 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:29.804+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:29.957 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:29.957+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:30.389 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:30.390+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:30.828 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:30.828+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:30.962 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:30.963+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:31.372 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:31.372+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:31.844 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:31.845+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:31.970 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:31.970+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:32.362 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:32.362+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:32.828 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:32.829+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:32.959 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:32.959+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:33.355 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:33.356+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:33.816 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:33.816+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:33.947 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:33.947+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:34.319 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:34.320+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:34.859 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:34.860+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:34.943 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:34.944+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:35.345 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:35.346+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:35.888 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:35.889+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:35.966 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:35.967+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:36.380 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:36.381+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:36.930 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:36.931+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:36.930 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:36.931+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:37.385 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:37.386+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:37.965 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:37.966+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:37.977 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:37.979+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:38.383 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:38.384+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:38.938 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:38.939+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:38.963 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:38.965+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:39.433 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:39.434+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:39.915 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:39.916+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:40.006 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:40.007+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:40.407 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:40.408+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:40.899 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:40.900+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:40.988 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:40.989+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:41.415 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:41.416+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:41.852 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:41.853+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:41.974 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:41.975+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:42.425 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:42.426+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:42.878 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:42.879+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:42.963 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:42.965+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:43.452 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:43.454+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:43.877 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:43.879+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:43.972 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:43.973+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:44.407 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:44.408+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:44.872 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:44.873+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:44.975 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:44.976+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:45.380 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:45.381+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:45.840 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:45.841+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:45.998 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:45.999+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:46.425 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:46.426+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:46.821 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:46.823+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:47.044 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:47.045+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:47.377 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:47.378+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:47.834 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:47.834+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:48.046 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:48.046+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:48.366 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:48.366+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:48.822 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:48.822+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:49.073 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:49.074+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:49.339 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:49.339+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:49.869 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:49.869+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:50.101 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:50.101+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:50.379 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:50.379+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:50.881 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:50.881+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:51.092 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:51.092+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:51.331 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:51.331+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:51.924 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:51.924+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:52.078 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:52.078+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:52.305 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:52.305+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:52.884 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:52.884+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:53.031 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:53.031+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:53.268 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:53.268+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:53.895 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:53.895+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:54.029 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:54.030+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:54.295 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:54.295+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:54.904 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:54.904+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:55.075 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:55.075+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:55.291 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:55.291+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:55.925 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:55.925+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:56.039 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:56.039+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:56.283 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:56.284+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:56.950 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:56.950+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:57.088 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:57.088+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:57.317 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:57.317+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:57.924 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:57.924+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:58.102 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:58.102+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:58.277 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:58.278+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:58.962 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:58.962+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:59.060 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:24:59.060+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:59.302 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:24:59.303+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:24:59.994 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:24:59.995+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:00.065 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:00.066+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:00.292 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:00.293+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:01.037 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:01.037+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:01.038 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:01.038+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:01.329 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:01.329+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:01.996 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:01.996+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:02.063 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:02.063+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:02.285 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:02.285+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:03.004 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:03.004+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:03.034 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:03.034+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:03.334 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:03.334+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:03.961 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:03.961+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:04.014 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:04.014+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:04.379 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:04.380+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:04.975 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:04.975+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:05.000 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:05.000+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:05.402 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:05.402+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:05.930 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:05.930+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:05.981 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:05.981+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:06.422 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:06.422+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:06.905 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:06.906+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:06.984 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:06.984+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:07.399 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:07.399+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:07.874 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:07.874+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:08.034 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:08.034+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:08.425 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:08.425+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:08.875 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:08.875+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:09.010 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:09.011+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:09.465 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:09.465+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:09.856 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:09.856+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:10.034 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:10.034+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:10.434 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:10.434+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:10.820 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:10.820+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:11.006 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:11.006+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:11.418 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:11.418+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:11.838 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:11.838+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:11.988 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:11.988+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:12.463 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:12.464+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:12.792 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:12.792+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:12.987 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:12.987+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:13.503 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:13.503+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:13.828 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:13.828+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:14.014 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:14.014+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:14.520 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:14.520+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:14.785 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:14.785+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:14.982 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:14.982+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:15.512 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:15.512+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:15.741 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:15.741+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:15.985 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:15.985+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:16.523 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:16.523+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:16.697 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:16.697+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:16.991 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:16.992+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:17.522 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:17.523+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:17.745 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:17.745+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:18.012 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:18.012+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:18.552 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:18.553+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:18.789 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:18.790+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:19.015 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:19.016+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:19.582 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:19.583+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:19.777 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:19.778+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:20.012 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:20.013+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:20.605 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:20.606+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:20.813 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:20.814+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:21.037 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:21.038+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:21.609 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:21.610+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:21.787 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:21.788+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:22.012 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:22.014+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:22.637 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:22.638+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:22.803 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:22.804+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:23.025 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:23.026+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:23.648 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:23.649+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:23.783 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:23.784+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:24.065 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:24.066+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:24.629 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:24.630+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:24.781 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:24.782+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:25.087 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:25.088+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:25.600 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:25.601+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:25.793 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:25.794+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:26.074 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:26.075+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:26.645 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:26.646+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:26.808 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:26.809+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:27.118 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:27.119+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:27.632 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:27.633+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:27.813 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:27.814+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:28.076 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:28.077+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:28.679 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:28.680+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:28.813 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:28.814+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:29.060 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:29.061+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:29.658 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:29.659+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:29.862 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:29.863+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:30.089 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:30.090+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:30.614 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:30.615+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:30.911 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:30.912+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:31.101 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:31.102+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:31.622 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:31.623+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:31.880 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:31.882+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:32.081 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:32.082+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:32.617 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:32.617+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:32.926 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:32.926+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:33.118 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:33.119+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:33.658 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:33.658+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:33.916 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:33.916+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:34.081 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:34.081+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:34.676 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:34.676+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:34.918 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:34.919+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:35.088 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:35.088+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:35.655 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:35.655+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:35.955 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:35.955+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:36.073 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:36.074+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:36.660 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:36.660+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:36.968 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:36.969+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:37.040 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:37.040+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:37.653 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:37.653+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:37.941 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:37.941+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:38.030 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:38.031+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:38.668 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:38.668+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:38.963 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:38.963+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:39.050 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:39.050+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:39.679 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:39.679+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:39.993 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:39.994+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:40.017 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:40.017+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:40.642 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:40.642+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:40.977 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:40.977+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:41.027 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:41.027+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:41.630 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:41.630+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:41.952 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:41.953+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:42.074 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:42.075+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:42.646 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:42.646+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:42.955 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:42.955+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:43.101 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:43.101+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:43.685 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:43.685+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:43.980 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:43.980+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:44.099 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:44.099+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:44.638 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:44.639+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:45.015 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:45.016+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:45.091 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:45.091+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:45.663 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:45.664+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:46.005 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:46.005+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:46.141 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:46.141+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:46.708 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:46.708+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:46.957 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:46.957+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:47.097 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:47.097+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:47.747 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:47.747+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:47.964 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:47.964+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:48.048 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:48.048+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:48.748 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:48.748+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:48.956 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:48.956+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:49.032 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:49.032+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:49.753 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:49.753+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:49.922 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:49.922+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:50.073 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:50.073+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:50.754 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:50.754+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:50.945 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:50.945+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:51.107 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:51.107+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:51.706 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:51.706+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:51.924 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:51.924+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:52.086 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:52.086+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:52.754 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:52.754+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:52.929 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:52.929+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:53.079 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:53.079+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:53.764 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:53.765+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:53.887 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:53.887+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:54.119 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:54.119+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:54.737 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:54.737+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:54.862 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:54.862+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:55.104 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:55.104+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:55.736 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:55.736+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:55.840 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:55.840+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:56.116 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:56.116+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:56.786 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:56.786+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:56.812 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:56.812+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:57.153 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:57.153+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:57.771 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:57.771+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:57.774 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:57.775+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:58.151 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:58.151+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:58.756 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:58.756+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:58.818 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:58.818+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:59.169 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:25:59.169+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:59.732 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:25:59.732+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:25:59.818 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:25:59.818+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:00.201 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:00.201+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:00.747 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:00.747+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:00.789 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:00.788+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:01.251 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:01.251+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:01.720 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:01.720+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:01.763 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:01.763+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:02.288 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:02.288+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:02.691 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:02.691+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:02.730 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:02.731+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:03.298 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:03.299+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:03.692 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:03.692+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:03.700 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:03.701+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:04.326 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:04.327+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:04.686 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:04.687+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:04.697 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:04.698+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:05.372 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:05.373+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:05.703 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:05.705+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:05.734 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:05.735+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:06.393 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:06.394+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:06.712 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:06.713+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:06.778 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:06.780+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:07.412 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:07.413+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:07.711 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:07.712+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:07.740 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:07.741+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:08.454 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:08.455+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:08.672 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:08.673+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:08.721 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:08.722+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:09.432 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:09.433+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:09.667 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:09.669+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:09.739 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:09.740+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:10.389 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:10.390+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:10.699 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:10.700+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:10.709 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:10.710+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:11.347 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:11.348+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:11.712 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:11.713+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:11.714 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:11.715+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:12.347 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:12.348+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:12.718 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:12.719+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:12.762 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:12.763+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:13.318 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:13.319+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:13.731 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:13.732+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:13.776 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:13.777+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:14.338 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:14.339+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:14.719 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:14.720+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:14.754 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:14.755+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:15.303 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:15.304+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:15.708 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:15.709+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:15.741 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:15.741+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:16.325 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:16.326+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:16.700 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:16.700+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:16.752 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:16.753+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:17.343 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:17.344+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:17.687 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:17.687+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:17.736 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:17.736+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:18.315 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:18.315+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:18.687 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:18.687+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:18.754 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:18.754+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:19.325 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:19.325+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:19.649 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:19.649+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:19.764 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:19.764+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:20.339 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:20.339+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:20.601 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:20.602+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:20.759 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:20.759+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:21.341 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:21.341+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:21.651 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:21.651+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:21.740 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:21.740+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:22.345 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:22.345+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:22.669 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:22.669+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:22.730 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:22.730+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:23.347 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:23.347+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:23.629 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:23.629+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:23.730 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:23.730+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:24.388 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:24.388+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:24.595 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:24.596+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:24.769 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:24.769+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:25.373 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:25.373+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:25.546 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:25.546+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:25.764 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:25.764+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:26.337 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:26.336+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:26.592 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:26.592+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:26.784 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:26.784+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:27.328 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:27.328+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:27.599 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:27.599+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:27.740 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:27.740+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:28.288 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:28.288+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:28.604 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:28.604+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:28.692 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:28.692+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:29.267 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:29.267+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:29.557 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:29.557+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:29.683 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:29.683+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:30.266 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:30.266+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:30.582 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:30.582+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:30.700 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:30.700+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:31.262 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:31.262+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:31.618 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:31.618+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:31.746 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:31.746+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:32.243 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:32.243+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:32.644 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:32.644+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:32.728 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:32.728+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:33.266 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:33.266+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:33.674 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:33.674+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:33.700 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:33.700+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:34.270 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:34.270+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:34.672 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:34.673+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:34.674 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:34.674+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:35.304 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:35.304+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:35.687 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:35.687+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:35.713 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:35.713+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:36.318 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:36.318+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:36.689 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:36.689+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:36.695 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:36.695+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:37.359 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:37.360+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:37.644 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:37.644+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:37.719 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:37.719+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:38.318 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:38.318+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:38.675 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:38.675+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:38.738 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:38.738+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:39.307 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:39.307+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:39.674 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:39.674+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:39.713 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:39.714+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:40.340 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:40.340+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:40.686 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:40.686+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:40.740 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:40.740+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:41.338 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:41.338+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:41.700 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:41.700+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:41.704 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:41.704+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:42.339 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:42.339+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:42.703 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:42.703+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:42.742 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:42.742+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:43.360 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:43.360+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:43.663 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:43.663+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:43.752 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:43.752+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:44.384 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:44.384+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:44.687 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:44.687+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:44.731 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:44.732+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:45.368 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:45.368+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:45.679 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:45.679+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:45.774 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:45.774+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:46.358 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:46.358+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:46.700 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:46.700+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:46.746 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:46.746+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:47.397 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:47.398+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:47.729 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:47.729+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:47.735 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:47.736+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:48.400 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:48.400+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:48.709 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:48.709+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:48.723 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:48.724+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:49.444 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:49.445+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:49.693 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:49.695+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:49.705 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:49.706+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:50.402 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:50.403+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:50.674 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:50.675+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:50.686 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:50.688+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:51.383 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:51.384+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:51.672 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:51.673+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:51.727 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:51.728+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:52.415 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:52.415+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:52.630 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:52.631+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:52.713 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:52.714+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:53.371 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:53.372+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:53.675 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:53.676+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:53.723 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:53.724+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:54.356 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:54.357+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:54.699 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:54.700+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:54.766 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:54.767+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:55.319 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:55.320+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:55.723 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:55.724+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:55.807 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:55.808+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:56.301 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:56.302+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:56.773 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:56.774+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:56.855 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:56.856+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:57.277 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:57.278+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:57.741 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:57.742+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:57.891 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:57.892+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:58.300 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:58.301+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:58.748 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:58.749+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:58.860 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:58.861+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:59.263 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:26:59.264+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:59.725 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:26:59.726+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:26:59.907 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:26:59.908+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:00.242 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:00.243+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:00.757 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:00.758+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:00.860 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:00.861+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:01.200 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:01.201+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:01.734 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:01.735+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:01.822 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:01.823+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:02.213 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:02.211+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:02.700 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:02.700+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:02.826 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:02.822+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:03.209 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:03.209+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:03.681 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:03.681+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:03.871 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:03.871+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:04.246 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:04.247+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:04.719 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:04.719+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:04.874 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:04.874+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:05.282 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:05.282+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:05.765 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:05.765+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:05.849 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:05.849+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:06.327 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:06.327+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:06.733 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:06.733+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:06.890 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:06.890+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:07.337 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:07.337+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:07.749 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:07.749+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:07.886 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:07.886+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:08.349 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:08.348+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:08.735 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:08.735+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:08.906 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:08.906+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:09.323 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:09.323+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:09.741 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:09.741+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:09.910 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:09.910+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:10.368 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:10.368+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:10.755 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:10.755+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:10.953 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:10.953+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:11.354 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:11.354+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:11.713 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:11.713+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:11.915 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:11.915+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:12.386 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:12.386+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:12.688 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:12.689+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:12.941 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:12.941+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:13.343 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:13.343+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:13.734 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:13.734+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:13.980 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:13.980+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:14.309 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:14.310+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:14.707 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:14.707+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:15.017 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:15.017+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:15.340 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:15.340+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:15.679 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:15.679+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:16.009 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:16.009+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:16.340 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:16.340+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:16.691 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:16.691+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:17.003 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:17.003+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:17.390 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:17.390+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:17.732 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:17.732+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:17.972 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:17.972+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:18.369 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:18.369+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:18.704 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:18.704+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:18.954 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:18.954+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:19.357 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:19.357+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:19.726 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:19.726+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:19.956 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:19.956+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:20.323 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:20.323+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:20.731 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:20.731+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:20.938 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:20.938+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:21.327 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:21.327+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:21.775 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:21.766+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:21.944 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:21.944+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:22.324 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:22.325+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:22.797 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:22.797+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:22.977 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:22.978+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:23.316 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:23.316+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:23.784 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:23.784+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:23.970 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:23.969+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:24.335 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:24.335+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:24.781 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:24.781+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:24.947 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:24.947+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:25.332 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:25.332+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:25.757 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:25.757+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:25.981 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:25.981+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:26.294 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:26.294+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:26.802 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:26.803+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:27.016 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:27.016+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:27.342 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:27.342+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:27.814 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:27.815+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:27.969 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:27.969+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:28.302 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:28.302+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:28.835 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:28.835+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:28.960 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:28.960+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:29.351 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:29.351+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:29.829 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:29.829+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:29.994 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:29.994+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:30.347 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:30.347+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:30.874 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:30.874+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:30.962 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:30.962+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:31.330 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:31.330+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:31.870 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:31.870+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:31.953 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:31.953+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:32.283 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:32.283+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:32.847 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:32.847+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:32.911 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:32.912+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:33.319 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:33.319+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:33.836 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:33.836+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:33.903 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:33.903+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:34.320 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:34.321+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:34.869 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:34.870+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:34.876 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:34.877+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:35.353 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:35.354+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:35.901 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:35.902+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:35.917 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:35.918+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:36.321 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:36.322+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:36.879 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:36.880+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:36.968 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:36.969+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:37.316 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:37.317+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:37.883 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:37.884+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:37.987 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:37.988+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:38.284 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:38.285+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:38.921 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:38.922+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:39.009 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:39.010+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:39.298 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:39.299+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:39.881 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:39.882+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:39.969 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:39.970+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:40.336 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:40.337+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:40.905 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:40.906+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:40.991 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:40.992+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:41.291 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:41.292+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:41.927 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:41.928+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:41.950 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:41.951+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:42.267 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:42.268+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:42.960 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:42.961+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:42.984 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:42.985+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:43.280 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:43.281+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:43.924 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:43.925+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:43.970 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:43.971+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:44.286 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:44.288+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:44.959 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:44.960+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:44.963 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:44.964+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:45.244 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:45.245+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:45.946 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:45.947+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:45.948 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:45.949+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:46.258 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:46.259+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:46.962 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:46.963+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:46.967 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:46.968+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:47.308 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:47.309+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:47.928 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:47.928+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:47.932 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:47.932+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:48.330 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:48.330+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:48.894 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:48.894+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:48.902 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:48.902+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:49.333 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:49.333+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:49.925 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:49.925+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:49.928 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:49.929+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:50.374 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:50.374+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:50.931 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:50.931+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:50.972 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:50.972+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:51.343 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:51.343+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:51.928 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:51.928+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:51.968 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:51.968+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:52.339 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:52.339+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:52.920 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:52.920+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:52.944 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:52.944+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:53.365 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:53.365+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:53.905 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:53.905+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:53.960 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:53.960+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:54.341 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:54.341+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:54.878 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:54.878+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:54.936 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:54.936+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:55.292 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:55.292+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:55.835 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:55.835+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:55.970 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:55.970+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:56.246 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:56.246+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:56.802 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:56.802+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:57.004 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:57.004+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:57.276 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:57.276+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:57.792 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:57.793+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:58.021 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:58.021+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:58.234 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:58.234+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:58.751 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:58.751+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:59.060 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:27:59.060+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:59.190 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:27:59.190+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:27:59.791 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:27:59.791+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:00.054 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:00.054+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:00.224 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:00.224+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:00.785 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:00.784+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:01.010 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:01.010+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:01.206 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:01.206+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:01.768 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:01.768+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:02.045 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:02.042+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:02.163 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:02.163+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:02.763 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:02.763+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:03.034 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:03.034+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:03.188 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:03.188+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:03.717 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:03.717+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:04.000 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:04.000+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:04.150 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:04.150+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:04.709 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:04.709+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:05.043 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:05.043+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:05.107 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:05.107+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:05.681 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:05.680+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:06.085 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:06.085+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:06.086 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:06.087+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:06.677 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:06.677+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:07.064 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:07.064+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:07.092 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:07.092+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:07.710 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:07.710+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:08.046 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:08.046+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:08.112 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:08.112+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:08.736 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:08.736+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:09.045 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:09.045+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:09.163 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:09.162+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:09.723 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:09.723+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:10.029 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:10.029+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:10.126 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:10.126+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:10.673 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:10.673+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:10.982 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:10.982+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:11.122 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:11.122+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:11.638 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:11.638+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:11.941 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:11.942+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:12.108 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:12.108+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:12.654 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:12.654+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:12.951 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:12.951+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:13.102 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:13.102+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:13.684 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:13.684+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:13.980 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:13.980+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:14.109 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:14.109+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:14.719 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:14.719+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:15.028 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:15.028+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:15.156 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:15.156+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:15.689 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:15.689+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:16.046 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:16.046+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:16.184 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:16.184+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:16.644 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:16.645+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:17.059 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:17.060+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:17.198 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:17.198+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:17.624 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:17.625+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:18.071 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:18.071+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:18.201 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:18.201+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:18.637 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:18.639+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:19.101 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:19.101+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:19.216 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:19.217+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:19.649 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:19.650+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:20.074 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:20.075+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:20.227 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:20.227+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:20.612 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:20.613+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:21.063 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:21.064+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:21.203 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:21.204+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:21.580 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:21.581+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:22.023 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:22.024+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:22.225 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:22.226+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:22.573 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:22.574+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:23.065 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:23.066+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:23.213 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:23.214+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:23.527 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:23.528+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:24.049 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:24.050+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:24.238 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:24.240+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:24.497 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:24.498+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:25.017 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:25.018+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:25.263 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:25.264+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:25.540 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:25.541+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:26.063 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:26.064+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:26.276 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:26.277+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:26.549 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:26.550+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:27.070 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:27.071+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:27.316 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:27.317+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:27.543 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:27.544+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:28.045 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:28.046+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:28.318 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:28.318+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:28.537 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:28.538+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:29.025 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:29.026+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:29.367 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:29.368+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:29.501 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:29.502+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:30.065 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:30.066+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:30.327 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:30.328+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:30.460 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:30.461+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:31.016 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:31.017+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:31.375 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:31.376+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:31.422 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:31.423+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:31.993 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:31.994+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:32.403 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:32.403+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:32.435 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:32.435+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:32.987 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:32.987+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:33.392 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:33.393+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:33.403 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:33.403+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:34.033 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:34.033+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:34.368 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:34.368+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:34.420 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:34.420+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:35.021 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:35.021+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:35.342 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:35.342+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:35.393 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:35.393+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:35.986 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:35.985+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:36.313 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:36.313+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:36.431 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:36.431+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:37.033 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:37.033+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:37.332 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:37.331+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:37.437 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:37.437+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:37.985 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:37.985+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:38.304 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:38.304+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:38.423 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:38.423+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:38.975 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:38.975+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:39.335 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:39.334+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:39.454 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:39.454+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:39.983 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:39.983+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:40.372 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:40.372+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:40.503 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:40.501+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:41.023 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:41.023+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:41.364 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:41.364+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:41.487 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:41.487+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:42.025 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:42.025+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:42.410 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:42.410+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:42.510 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:42.510+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:43.017 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:43.017+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:43.411 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:43.411+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:43.559 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:43.559+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:44.022 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:44.022+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:44.423 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:44.423+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:44.538 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:44.538+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:45.007 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:45.007+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:45.456 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:45.456+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:45.504 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:45.504+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:45.981 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:45.981+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:46.501 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:46.501+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:46.543 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:46.543+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:46.938 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:46.939+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:47.475 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:47.475+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:47.517 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:47.517+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:47.964 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:47.964+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:48.491 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:48.491+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:48.535 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:48.535+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:48.956 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:48.956+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:49.483 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:49.483+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:49.557 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:49.557+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:49.936 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:49.936+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:50.504 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:50.503+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:50.531 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:50.531+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:50.897 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:50.897+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:51.511 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:51.511+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:51.512 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:51.512+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:51.899 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:51.899+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:52.484 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:52.484+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:52.537 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:52.537+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:52.919 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:52.919+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:53.493 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:53.491+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:53.554 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:53.554+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:53.887 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:53.887+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:54.531 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:54.531+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:54.579 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:54.579+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:54.894 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:54.894+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:55.571 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:55.571+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:55.593 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:55.593+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:55.864 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:55.864+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:56.552 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:56.552+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:56.554 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:56.554+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:56.869 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:56.869+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:57.570 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:57.570+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:57.588 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:57.588+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:57.889 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:57.889+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:58.535 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:58.535+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:58.614 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:58.614+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:58.841 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:58.841+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:59.582 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:28:59.582+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:59.632 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:28:59.632+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:28:59.887 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:28:59.887+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:00.605 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:00.605+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:00.645 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:00.645+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:00.899 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:00.899+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:01.628 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:01.628+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:01.654 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:01.654+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:01.863 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:01.863+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:02.612 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:02.612+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:02.650 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:02.650+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:02.873 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:02.873+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:03.623 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:03.622+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:03.683 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:03.683+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:03.861 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:03.861+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:04.651 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:04.652+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:04.659 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:04.660+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:04.830 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:04.831+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:05.682 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:05.683+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:05.695 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:05.696+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:05.833 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:05.834+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:06.637 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:06.638+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:06.646 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:06.647+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:06.852 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:06.853+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:07.600 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:07.601+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:07.695 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:07.696+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:07.835 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:07.836+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:08.608 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:08.609+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:08.714 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:08.714+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:08.789 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:08.790+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:09.579 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:09.580+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:09.738 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:09.739+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:09.837 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:09.838+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:10.562 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:10.563+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:10.750 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:10.751+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:10.789 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:10.790+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:11.559 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:11.559+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:11.749 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:11.750+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:11.795 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:11.795+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:12.601 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:12.602+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:12.770 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:12.771+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:12.775 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:12.777+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:13.587 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:13.588+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:13.759 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:13.760+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:13.790 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:13.791+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:14.620 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:14.621+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:14.710 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:14.711+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:14.825 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:14.826+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:15.655 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:15.655+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:15.732 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:15.734+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:15.788 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:15.789+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:16.654 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:16.655+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:16.770 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:16.771+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:16.783 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:16.784+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:17.610 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:17.610+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:17.761 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:17.760+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:17.798 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:17.798+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:18.604 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:18.604+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:18.777 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:18.777+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:18.824 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:18.824+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:19.566 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:19.566+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:19.753 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:19.753+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:19.873 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:19.873+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:20.534 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:20.534+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:20.767 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:20.766+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:20.860 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:20.860+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:21.563 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:21.563+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:21.781 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:21.780+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:21.889 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:21.889+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:22.601 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:22.598+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:22.761 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:22.761+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:22.926 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:22.926+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:23.629 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:23.629+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:23.786 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:23.786+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:23.927 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:23.927+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:24.655 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:24.654+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:24.774 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:24.774+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:24.978 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:24.978+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:25.686 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:25.686+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:25.788 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:25.788+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:25.949 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:25.949+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:26.695 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:26.695+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:26.781 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:26.781+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:26.943 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:26.942+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:27.693 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:27.692+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:27.771 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:27.771+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:27.919 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:27.919+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:28.692 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:28.692+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:28.765 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:28.764+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:28.935 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:28.935+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:29.671 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:29.671+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:29.720 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:29.720+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:29.949 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:29.949+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:30.675 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:30.675+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:30.763 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:30.763+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:30.918 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:30.918+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:31.683 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:31.683+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:31.773 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:31.773+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:31.954 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:31.954+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:32.651 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:32.651+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:32.791 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:32.791+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:32.907 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:32.907+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:33.618 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:33.618+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:33.813 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:33.813+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:33.941 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:33.941+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:34.640 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:34.640+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:34.818 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:34.818+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:34.991 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:34.990+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:35.595 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:35.594+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:35.854 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:35.854+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:35.997 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:35.997+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:36.620 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:36.620+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:36.846 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:36.846+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:37.018 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:37.018+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:37.601 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:37.601+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:37.891 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:37.891+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:38.029 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:38.029+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:38.612 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:38.612+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:38.903 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:38.903+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:39.067 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:39.067+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:39.579 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:39.579+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:39.916 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:39.916+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:40.045 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:40.045+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:40.549 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:40.549+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:40.947 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:40.947+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:40.998 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:40.998+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:41.515 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:41.515+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:41.967 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:41.967+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:41.968 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:41.968+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:42.565 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:42.565+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:42.935 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:42.935+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:42.958 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:42.958+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:43.538 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:43.538+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:43.924 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:43.923+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:43.973 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:43.973+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:44.540 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:44.540+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:44.932 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:44.932+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:44.983 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:44.983+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:45.510 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:45.510+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:45.898 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:45.898+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:45.986 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:45.986+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:46.540 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:46.540+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:46.874 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:46.874+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:46.952 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:46.952+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:47.538 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:47.538+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:47.841 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:47.841+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:47.992 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:47.992+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:48.558 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:48.557+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:48.811 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:48.812+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:49.012 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:49.013+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:49.580 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:49.581+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:49.827 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:49.828+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:50.056 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:50.056+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:50.553 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:50.553+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:50.804 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:50.805+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:51.035 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:51.036+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:51.573 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:51.574+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:51.836 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:51.837+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:51.988 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:51.989+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:52.566 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:52.567+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:52.852 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:52.852+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:53.000 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:53.001+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:53.551 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:53.552+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:53.889 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:53.890+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:54.051 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:54.052+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:54.509 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:54.510+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:54.940 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:54.941+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:55.076 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:55.077+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:55.480 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:55.481+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:55.919 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:55.920+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:56.062 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:56.063+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:56.491 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:56.492+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:56.912 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:56.913+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:57.046 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:57.047+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:57.443 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:57.443+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:57.930 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:57.931+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:58.008 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:58.009+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:58.467 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:58.467+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:58.907 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:58.908+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:59.040 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:29:59.041+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:59.441 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:29:59.442+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:29:59.888 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:29:59.889+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:00.000 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:00.001+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:00.474 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:00.475+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:00.924 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:00.925+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:00.994 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:00.995+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:01.450 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:01.451+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:01.938 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:01.938+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:01.998 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:01.998+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:02.413 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:02.413+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:02.924 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:02.924+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:03.046 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:03.045+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:03.364 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:03.364+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:03.923 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:03.923+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:04.014 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:04.014+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:04.362 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:04.361+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:04.951 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:04.951+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:05.012 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:05.012+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:05.386 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:05.386+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:05.913 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:05.912+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:05.972 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:05.972+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:06.352 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:06.352+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:06.913 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:06.912+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:06.923 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:06.923+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:07.364 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:07.364+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:07.936 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:07.936+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:07.946 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:07.945+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:08.369 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:08.369+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:08.917 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:08.917+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:08.919 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:08.919+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:09.374 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:09.374+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:09.870 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:09.870+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:09.960 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:09.960+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:10.330 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:10.330+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:10.854 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:10.854+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:10.987 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:10.987+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:11.378 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:11.377+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:11.810 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:11.810+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:12.016 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:12.016+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:12.341 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:12.341+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:12.832 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:12.832+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:12.984 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:12.984+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:13.314 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:13.314+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:13.841 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:13.841+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:14.014 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:14.014+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:14.309 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:14.309+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:14.806 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:14.806+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:14.992 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:14.992+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:15.352 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:15.352+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:15.824 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:15.824+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:15.971 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:15.971+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:16.331 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:16.331+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:16.806 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:16.806+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:16.949 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:16.949+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:17.325 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:17.325+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:17.838 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:17.838+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:17.966 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:17.966+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:18.344 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:18.344+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:18.825 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:18.825+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:18.959 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:18.959+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:19.319 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:19.319+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:19.777 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:19.777+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:20.000 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:20.000+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:20.295 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:20.295+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:20.818 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:20.818+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:20.997 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:20.997+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:21.261 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:21.261+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:21.852 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:21.852+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:21.965 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:21.964+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:22.255 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:22.255+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:22.807 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:22.807+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:23.002 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:23.001+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:23.304 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:23.304+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:23.768 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:23.767+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:24.007 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:24.007+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:24.325 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:24.325+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:24.801 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:24.801+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:25.034 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:25.033+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:25.310 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:25.309+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:25.756 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:25.756+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:26.081 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:26.081+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:26.283 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:26.283+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:26.796 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:26.796+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:27.061 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:27.061+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:27.301 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:27.301+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:27.799 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:27.799+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:28.054 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:28.054+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:28.297 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:28.297+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:28.774 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:28.774+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:29.093 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:29.093+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:29.344 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:29.343+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:29.784 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:29.784+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:30.125 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:30.125+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:30.391 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:30.391+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:30.822 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:30.821+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:31.105 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:31.106+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:31.397 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:31.397+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:31.870 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:31.870+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:32.093 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:32.093+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:32.432 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:32.432+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:32.834 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:32.833+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:33.102 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:33.102+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:33.385 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:33.385+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:33.814 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:33.815+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:34.111 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:34.112+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:34.428 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:34.428+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:34.797 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:34.798+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:35.069 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:35.070+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:35.437 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:35.438+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:35.756 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:35.757+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:36.031 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:36.032+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:36.405 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:36.406+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:36.730 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:36.731+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:37.077 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:37.077+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:37.401 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:37.402+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:37.742 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:37.743+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:38.084 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:38.085+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:38.443 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:38.444+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:38.773 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:38.774+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:39.116 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:39.117+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:39.446 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:39.447+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:39.740 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:39.740+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:40.128 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:40.129+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:40.440 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:40.441+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:40.764 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:40.763+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:41.099 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:41.100+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:41.406 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:41.407+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:41.770 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:41.770+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:42.061 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:42.062+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:42.411 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:42.410+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:42.801 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:42.802+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:43.088 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:43.089+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:43.403 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:43.404+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:43.847 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:43.848+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:44.055 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:44.056+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:44.356 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:44.357+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:44.871 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:44.872+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:45.040 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:45.041+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:45.332 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:45.333+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:45.881 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:45.882+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:46.065 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:46.066+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:46.286 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:46.287+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:46.832 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:46.832+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:47.113 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:47.113+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:47.281 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:47.281+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:47.797 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:47.797+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:48.078 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:48.078+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:48.274 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:48.274+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:48.820 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:48.820+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:49.126 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:49.126+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:49.267 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:49.267+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:49.824 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:49.824+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:50.129 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:50.129+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:50.254 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:50.253+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:50.834 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:50.834+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:51.111 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:51.111+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:51.266 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:51.266+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:51.790 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:51.789+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:52.159 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:52.158+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:52.283 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:52.283+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:52.791 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:52.791+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:53.180 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:53.180+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:53.240 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:53.240+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:53.834 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:53.833+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:54.203 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:54.203+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:54.212 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:54.211+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:54.838 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:54.838+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:55.217 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:55.217+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:55.242 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:55.242+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:55.800 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:55.800+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:56.221 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:56.221+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:56.246 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:56.246+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:56.769 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:56.769+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:57.182 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:57.178+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:57.223 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:57.223+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:57.734 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:57.734+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:58.196 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:58.196+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:58.223 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:58.223+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:58.759 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:58.759+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:59.178 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:30:59.178+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:59.230 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:30:59.227+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:30:59.794 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:30:59.794+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:00.219 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:00.219+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:00.227 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:00.227+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:00.833 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:00.833+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:01.174 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:01.174+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:01.227 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:01.227+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:01.819 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:01.819+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:02.158 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:02.158+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:02.233 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:02.233+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:02.866 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:02.865+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:03.117 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:03.117+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:03.210 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:03.209+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:03.892 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:03.892+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:04.104 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:04.103+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:04.217 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:04.217+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:04.861 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:04.861+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:05.072 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:05.072+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:05.188 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:05.188+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:05.823 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:05.823+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:06.075 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:06.075+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:06.184 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:06.184+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:06.813 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:06.813+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:07.091 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:07.091+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:07.147 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:07.147+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:07.861 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:07.861+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:08.108 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:08.108+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:08.163 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:08.163+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:08.911 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:08.911+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:09.154 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:09.154+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:09.191 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:09.191+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:09.867 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:09.866+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:10.135 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:10.135+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:10.233 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:10.233+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:10.835 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:10.835+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:11.148 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:11.148+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:11.271 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:11.271+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:11.873 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:11.873+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:12.138 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:12.138+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:12.294 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:12.294+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:12.889 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:12.889+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:13.103 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:13.103+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:13.297 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:13.297+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:13.859 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:13.859+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:14.097 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:14.097+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:14.335 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:14.335+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:14.898 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:14.898+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:15.086 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:15.086+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:15.348 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:15.349+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:15.947 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:15.947+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:16.104 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:16.104+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:16.338 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:16.338+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:16.938 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:16.937+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:17.099 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:17.099+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:17.293 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:17.292+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:17.978 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:17.979+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:18.086 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:18.086+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:18.320 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:18.321+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:18.935 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:18.936+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:19.077 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:19.078+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:19.352 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:19.353+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:19.984 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:19.985+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:20.062 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:20.063+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:20.341 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:20.342+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:21.013 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:21.014+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:21.040 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:21.041+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:21.348 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:21.349+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:21.974 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:21.975+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:22.083 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:22.084+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:22.343 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:22.344+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:22.970 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:22.971+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:23.059 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:23.060+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:23.322 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:23.323+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:23.966 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:23.967+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:24.095 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:24.096+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:24.355 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:24.356+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:24.987 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:24.988+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:25.128 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:25.129+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:25.365 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:25.365+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:26.030 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:26.028+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:26.129 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:26.130+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:26.365 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:26.366+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:27.064 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:27.065+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:27.156 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:27.157+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:27.326 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:27.326+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:28.086 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:28.087+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:28.117 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:28.117+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:28.285 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:28.286+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:29.074 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:29.075+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:29.137 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:29.137+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:29.316 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:29.317+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:30.094 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:30.095+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:30.151 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:30.152+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:30.283 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:30.284+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:31.131 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:31.132+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:31.132 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:31.134+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:31.257 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:31.257+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:32.105 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:32.105+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:32.105 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:32.105+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:32.218 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:32.218+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:33.064 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:33.064+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:33.133 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:33.133+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:33.192 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:33.192+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:34.026 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:34.026+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:34.110 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:34.110+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:34.211 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:34.211+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:35.067 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:35.067+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:35.076 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:35.075+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:35.230 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:35.230+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:36.059 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:36.058+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:36.097 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:36.096+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:36.206 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:36.206+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:37.090 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:37.090+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:37.102 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:37.101+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:37.192 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:37.192+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:38.104 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:38.104+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:38.129 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:38.129+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:38.236 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:38.235+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:39.057 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:39.057+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:39.108 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:39.108+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:39.241 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:39.241+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:40.018 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:40.018+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:40.150 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:40.150+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:40.288 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:40.288+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:40.986 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:40.986+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:41.116 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:41.116+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:41.255 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:41.255+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:42.036 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:42.036+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:42.153 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:42.152+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:42.228 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:42.228+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:43.084 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:43.084+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:43.184 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:43.184+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:43.193 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:43.193+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:44.129 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:44.128+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:44.178 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:44.178+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:44.236 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:44.235+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:45.150 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:45.149+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:45.176 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:45.176+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:45.261 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:45.261+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:46.155 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:46.155+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:46.156 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:46.156+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:46.252 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:46.252+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:47.122 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:47.121+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:47.203 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:47.203+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:47.268 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:47.267+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:48.157 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:48.156+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:48.190 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:48.190+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:48.264 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:48.264+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:49.176 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:49.176+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:49.205 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:49.205+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:49.304 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:49.304+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:50.198 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:50.197+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:50.206 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:50.206+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:50.288 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:50.288+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:51.183 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:51.182+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:51.183 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:51.183+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:51.270 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:51.270+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:52.143 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:52.142+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:52.183 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:52.183+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:52.319 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:52.318+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:53.113 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:53.112+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:53.178 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:53.177+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:53.324 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:53.324+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:54.083 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:54.083+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:54.181 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:54.181+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:54.354 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:54.354+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:55.036 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:55.036+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:55.221 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:55.221+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:55.325 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:55.325+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:56.079 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:56.078+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:56.249 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:56.249+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:56.302 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:56.302+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:57.128 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:57.128+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:57.212 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:57.212+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:57.264 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:57.263+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:58.123 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:58.123+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:58.227 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:58.227+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:58.290 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:58.290+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:59.074 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:31:59.074+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:59.203 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:31:59.203+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:31:59.335 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:31:59.335+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:00.039 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:00.040+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:00.177 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:00.178+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:00.342 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:00.342+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:01.016 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:01.016+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:01.211 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:01.212+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:01.310 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:01.310+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:02.050 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:02.050+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:02.215 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:02.214+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:02.314 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:02.314+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:03.085 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:03.086+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:03.176 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:03.177+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:03.302 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:03.303+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:04.064 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:04.065+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:04.168 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:04.169+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:04.258 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:04.259+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:05.020 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:05.021+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:05.195 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:05.196+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:05.302 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:05.303+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:06.035 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:06.035+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:06.154 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:06.155+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:06.290 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:06.290+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:07.022 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:07.023+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:07.188 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:07.189+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:07.312 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:07.313+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:08.019 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:08.020+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:08.176 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:08.176+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:08.275 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:08.276+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:08.977 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:08.978+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:09.210 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:09.211+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:09.246 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:09.247+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:09.968 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:09.969+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:10.217 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:10.218+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:10.293 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:10.293+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:10.970 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:10.971+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:11.233 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:11.234+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:11.302 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:11.303+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:11.934 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:11.935+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:12.257 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:12.258+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:12.265 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:12.266+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:12.968 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:12.969+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:13.225 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:13.226+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:13.240 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:13.241+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:13.982 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:13.983+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:14.194 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:14.195+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:14.233 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:14.234+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:15.031 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:15.032+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:15.165 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:15.166+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:15.215 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:15.215+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:16.000 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:16.000+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:16.202 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:16.202+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:16.245 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:16.245+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:16.965 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:16.965+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:17.175 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:17.174+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:17.247 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:17.247+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:17.921 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:17.921+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:18.166 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:18.166+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:18.208 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:18.208+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:18.916 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:18.916+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:19.166 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:19.166+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:19.218 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:19.218+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:19.908 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:19.908+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:20.200 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:20.200+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:20.239 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:20.239+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:20.921 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:20.921+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:21.163 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:21.163+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:21.222 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:21.222+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:21.882 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:21.882+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:22.126 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:22.126+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:22.245 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:22.245+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:22.896 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:22.894+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:23.128 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:23.128+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:23.264 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:23.264+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:23.847 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:23.847+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:24.133 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:24.133+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:24.292 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:24.291+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:24.803 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:24.802+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:25.150 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:25.150+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:25.328 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:25.328+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:25.762 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:25.762+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:26.134 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:26.134+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:26.354 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:26.353+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:26.800 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:26.800+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:27.183 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:27.182+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:27.348 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:27.348+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:27.763 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:27.763+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:28.164 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:28.164+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:28.299 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:28.299+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:28.806 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:28.805+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:29.200 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:29.200+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:29.252 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:29.251+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:29.814 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:29.813+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:30.164 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:30.164+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:30.255 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:30.255+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:30.786 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:30.786+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:31.203 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:31.203+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:31.240 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:31.240+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:31.823 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:31.823+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:32.188 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:32.188+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:32.247 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:32.246+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:32.813 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:32.813+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:33.202 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:33.202+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:33.235 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:33.235+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:33.820 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:33.819+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:34.220 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:34.220+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:34.226 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:34.226+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:34.856 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:34.856+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:35.229 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:35.229+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:35.251 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:35.250+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:35.902 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:35.902+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:36.250 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:36.249+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:36.250 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:36.250+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:36.928 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:36.927+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:37.226 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:37.225+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:37.272 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:37.272+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:37.890 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:37.890+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:38.260 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:38.259+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:38.284 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:38.284+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:38.867 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:38.867+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:39.265 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:39.265+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:39.312 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:39.312+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:39.882 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:39.882+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:40.276 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:40.275+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:40.363 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:40.363+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:40.908 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:40.908+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:41.297 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:41.296+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 2 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:41.389 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:41.389+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:41.868 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:41.868+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:42.262 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:42.261+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:42.346 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:42.346+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:42.357 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:32:42.357+0000 7fccf53da700 0 -- 172.21.15.87:0/3193322219 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x564162b40680 msgr2=0x7fccd8036370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-24T18:32:42.901 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:42.901+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:43.311 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:43.311+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:43.315 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:43.315+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:43.886 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:43.886+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:44.291 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:44.291+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:44.326 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:44.325+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:44.917 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:44.917+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:45.278 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:45.278+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:45.347 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:45.347+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:45.913 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:45.912+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:46.236 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:46.236+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:46.375 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:46.375+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:46.882 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:46.881+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:47.266 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:47.266+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:47.426 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:47.425+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:47.840 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:47.841+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:48.240 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:48.241+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:48.458 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:48.458+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:48.802 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:48.803+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:49.280 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:49.281+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:49.430 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:49.431+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:49.815 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:49.816+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:50.273 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:50.274+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:50.470 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:50.471+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:50.790 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:50.791+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:51.301 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:51.302+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:51.517 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:51.518+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:51.747 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:51.748+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:52.278 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:52.279+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:52.490 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:52.490+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:52.731 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:52.731+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:53.299 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:53.299+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:53.473 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:53.474+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:53.723 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:53.724+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:54.265 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:54.265+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:54.490 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:54.490+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:54.759 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:54.760+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:55.226 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:55.227+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:55.534 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:55.535+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:55.771 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:55.772+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:56.204 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:56.205+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:56.516 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:56.517+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:56.812 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:56.812+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:57.251 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:57.252+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:57.485 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:57.486+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:57.841 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:57.841+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:58.249 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:58.249+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:58.447 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:58.447+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:58.811 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:58.812+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:59.242 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:32:59.242+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:59.441 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:32:59.439+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:32:59.842 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:32:59.843+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:00.251 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:00.252+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:00.434 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:00.434+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:00.816 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:00.815+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:01.249 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:01.249+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:01.468 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:01.468+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:01.782 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:01.781+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:02.250 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:02.250+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:02.452 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:02.452+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:02.758 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:02.757+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:03.254 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:03.253+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:03.483 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:03.483+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:03.720 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:03.720+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:04.210 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:04.210+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:04.495 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:04.495+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:04.754 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:04.754+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:05.235 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:05.235+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:05.519 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:05.519+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:05.740 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:05.740+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:06.232 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:06.232+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:06.543 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:06.542+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:06.724 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:06.724+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:07.220 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:07.220+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:07.501 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:07.500+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:07.715 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:07.715+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:08.246 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:08.245+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:08.543 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:08.543+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:08.697 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:08.697+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:09.227 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:09.227+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:09.577 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:09.576+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:09.729 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:09.729+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:10.238 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:10.238+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:10.551 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:10.551+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:10.711 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:10.711+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:11.286 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:11.286+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:11.600 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:11.600+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:11.719 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:11.719+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:12.312 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:12.312+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:12.646 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:12.646+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:12.743 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:12.743+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:13.310 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:13.309+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:13.605 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:13.605+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:13.744 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:13.744+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:14.283 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:14.283+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:14.573 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:14.572+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:14.719 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:14.719+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:15.259 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:15.258+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:15.547 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:15.546+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:15.741 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:15.741+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:16.304 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:16.304+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:16.576 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:16.576+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:16.730 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:16.730+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:17.333 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:17.333+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:17.529 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:17.529+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:17.753 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:17.752+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:18.379 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:18.379+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:18.534 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:18.534+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:18.774 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:18.773+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:19.400 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:19.400+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:19.544 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:19.543+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:19.760 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:19.760+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:20.382 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:20.382+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:20.580 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:20.580+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:20.783 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:20.783+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:21.385 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:21.385+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:21.533 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:21.533+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:21.773 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:21.773+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:22.379 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:22.379+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:22.557 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:22.556+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:22.805 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:22.805+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:23.427 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:23.427+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:23.523 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:23.523+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:23.793 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:23.793+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:24.455 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:24.455+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:24.493 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:24.493+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:24.763 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:24.763+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:25.453 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:25.453+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:25.461 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:25.461+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:25.789 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:25.789+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:26.460 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:26.460+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:26.463 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:26.463+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:26.823 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:26.823+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:27.461 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:27.461+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:27.509 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:27.509+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:27.775 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:27.775+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:28.440 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:28.440+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:28.508 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:28.508+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:28.815 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:28.815+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:29.458 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:29.458+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:29.551 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:29.551+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:29.823 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:29.822+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:30.485 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:30.485+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:30.552 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:30.551+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:30.816 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:30.816+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:31.513 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:31.513+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:31.532 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:31.532+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:31.788 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:31.788+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:32.484 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:32.483+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:32.525 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:32.525+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:32.831 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:32.831+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:33.531 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:33.532+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:33.538 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:33.539+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:33.862 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:33.862+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:34.522 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:34.522+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:34.567 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:34.568+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:34.821 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:34.822+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:35.488 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:35.489+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:35.613 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:35.614+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:35.831 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:35.832+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:36.442 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:36.443+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:36.660 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:36.661+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:36.804 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:36.805+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:37.398 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:37.398+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:37.633 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:37.634+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:37.804 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:37.805+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:38.436 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:38.437+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:38.619 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:38.620+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:38.811 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:38.812+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:39.410 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:39.410+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:39.611 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:39.612+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:39.772 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:39.773+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:40.374 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:40.375+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:40.597 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:40.598+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:40.734 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:40.735+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:41.340 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:41.341+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:41.595 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:41.596+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:41.782 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:41.783+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:42.373 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:42.373+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:42.618 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:42.619+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:42.772 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:42.773+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:43.420 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:43.421+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:43.576 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:43.576+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:43.730 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:43.731+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:44.376 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:44.376+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:44.607 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:44.607+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:44.763 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:44.764+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:45.389 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:45.389+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:45.563 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:45.564+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:45.754 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:45.753+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:46.340 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:46.340+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:46.533 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:46.533+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:46.742 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:46.741+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:47.332 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:47.332+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:47.544 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:47.544+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:47.699 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:47.698+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:48.360 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:48.360+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:48.557 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:48.556+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:48.690 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:48.690+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:49.361 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:49.360+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:49.561 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:49.560+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:49.728 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:49.728+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:50.392 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:50.392+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:50.515 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:50.515+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:50.755 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:50.754+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:51.365 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:51.365+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:51.560 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:51.559+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:51.755 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:51.755+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:52.374 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:52.373+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:52.552 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:52.552+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:52.773 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:52.773+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:53.332 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:53.332+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:53.578 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:53.578+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:53.786 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:53.785+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:54.373 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:54.373+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:54.608 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:54.607+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:54.738 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:54.738+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:55.324 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:55.324+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:55.647 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:55.647+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:55.772 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:55.772+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:56.290 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:56.290+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:56.670 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:56.670+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:56.794 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:56.794+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:57.249 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:57.249+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:57.683 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:57.683+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:57.774 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:57.774+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:58.242 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:58.242+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:58.669 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:58.669+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:58.766 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:58.766+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:59.290 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:33:59.290+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:59.713 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:33:59.713+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:33:59.747 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:33:59.747+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:00.279 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:00.279+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:00.707 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:00.707+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:00.719 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:00.719+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:01.308 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:01.307+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:01.708 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:01.708+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:01.763 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:01.763+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:02.261 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:02.261+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:02.729 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:02.728+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:02.742 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:02.742+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:03.276 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:03.276+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:03.762 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:03.761+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:03.786 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:03.786+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:04.269 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:04.269+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:04.742 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:04.742+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:04.785 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:04.785+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:05.271 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:05.271+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:05.709 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:05.709+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:05.774 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:05.774+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:06.297 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:06.297+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:06.682 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:06.682+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:06.751 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:06.751+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:07.270 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:07.270+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:07.724 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:07.723+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:07.770 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:07.770+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:08.304 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:08.304+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:08.728 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:08.728+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:08.738 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:08.737+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:09.323 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:09.323+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:09.747 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:09.746+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:09.784 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:09.784+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:10.335 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:10.334+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:10.769 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:10.769+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:10.778 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:10.778+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:11.368 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:11.368+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:11.760 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:11.759+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:11.760 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:11.760+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:12.403 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:12.403+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:12.726 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:12.725+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:12.735 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:12.735+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:13.380 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:13.379+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:13.726 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:13.725+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:13.737 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:13.737+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:14.393 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:14.392+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:14.701 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:14.701+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:14.703 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:14.703+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:15.373 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:15.372+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:15.675 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:15.674+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:15.751 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:15.752+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:16.394 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:16.394+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:16.721 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:16.721+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:16.791 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:16.791+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:17.387 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:17.387+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:17.746 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:17.747+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:17.840 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:17.841+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:18.424 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:18.425+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:18.737 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:18.738+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:18.865 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:18.866+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:19.468 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:19.468+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:19.709 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:19.709+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:19.908 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:19.908+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:20.492 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:20.492+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:20.726 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:20.727+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:20.939 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:20.939+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:21.473 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:21.474+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:21.689 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:21.690+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:21.919 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:21.920+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:22.507 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:22.508+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:22.694 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:22.695+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:22.901 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:22.902+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:23.475 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:23.475+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:23.647 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:23.648+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:23.883 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:23.884+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:24.513 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:24.514+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:24.608 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:24.608+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:24.931 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:24.931+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:25.529 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:25.530+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:25.627 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:25.628+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:25.916 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:25.917+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:26.553 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:26.554+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:26.600 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:26.601+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:26.872 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:26.873+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:27.552 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:27.553+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:27.596 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:27.597+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:27.920 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:27.920+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:28.587 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:28.588+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:28.589 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:28.590+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:28.892 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:28.893+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:29.587 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:29.588+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:29.609 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:29.610+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:29.927 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:29.928+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:30.597 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:30.597+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:30.607 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:30.607+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:30.913 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:30.912+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:31.613 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:31.613+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:31.634 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:31.633+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:31.938 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:31.938+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:32.662 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:32.662+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:32.671 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:32.671+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:32.969 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:32.968+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:33.619 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:33.619+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:33.668 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:33.668+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:34.006 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:34.005+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:34.576 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:34.575+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:34.626 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:34.625+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:34.959 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:34.959+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:35.624 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:35.624+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:35.653 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:35.653+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:35.924 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:35.923+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:36.629 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:36.628+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:36.658 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:36.658+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:36.970 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:36.970+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:37.613 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:37.612+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:37.625 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:37.625+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:38.008 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:38.008+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:38.599 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:38.599+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:38.623 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:38.623+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:38.959 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:38.959+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:39.649 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:39.649+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:39.668 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:39.668+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:39.973 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:39.973+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:40.625 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:40.625+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:40.717 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:40.716+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:40.981 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:40.981+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:41.583 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:41.583+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:41.725 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:41.724+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:41.966 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:41.966+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:42.567 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:42.567+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:42.716 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:42.716+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:42.964 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:42.964+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:43.608 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:43.607+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:43.733 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:43.732+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:43.962 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:43.961+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:44.583 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:44.583+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:44.774 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:44.774+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:44.924 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:44.923+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:45.565 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:45.565+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:45.756 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:45.756+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:45.900 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:45.900+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:46.601 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:46.601+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:46.773 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:46.773+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:46.876 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:46.876+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:47.597 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:47.597+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:47.744 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:47.744+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:47.894 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:47.894+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:48.607 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:48.606+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:48.721 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:48.721+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:48.919 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:48.919+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:49.559 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:49.559+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:49.740 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:49.739+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:49.900 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:49.899+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:50.568 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:50.568+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:50.692 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:50.691+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:50.899 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:50.898+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:51.581 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:51.581+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:51.671 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:51.671+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:51.925 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:51.924+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:52.562 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:52.562+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:52.642 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:52.641+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:52.878 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:52.878+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:53.605 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:53.604+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:53.691 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:53.691+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:53.862 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:53.861+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:54.565 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:54.563+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:54.691 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:54.690+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:54.849 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:54.848+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:55.542 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:55.542+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:55.730 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:55.730+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:55.853 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:55.852+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:56.559 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:56.559+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:56.775 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:56.775+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:56.823 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:56.822+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:57.565 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:57.565+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:57.729 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:57.729+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:57.813 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:57.813+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:58.581 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:58.580+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:58.709 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:58.708+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:58.824 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:58.823+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:59.578 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:34:59.578+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:59.686 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:34:59.685+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:34:59.865 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:34:59.866+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:00.575 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:00.575+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:00.647 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:00.646+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:00.824 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:00.824+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:01.611 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:01.611+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:01.621 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:01.621+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:01.852 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:01.853+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:02.573 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:02.574+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:02.604 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:02.605+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:02.823 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:02.823+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:03.579 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:03.580+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:03.597 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:03.598+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:03.851 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:03.852+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:04.609 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:04.610+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:04.619 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:04.620+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:04.804 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:04.804+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:05.575 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:05.575+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:05.657 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:05.656+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:05.780 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:05.781+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:06.603 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:06.604+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:06.636 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:06.637+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:06.775 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:06.776+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:07.620 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:07.621+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:07.630 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:07.630+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:07.733 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:07.733+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:08.593 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:08.594+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:08.668 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:08.669+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:08.745 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:08.746+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:09.595 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:09.595+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:09.667 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:09.668+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:09.785 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:09.785+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:10.635 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:10.636+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:10.658 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:10.658+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:10.807 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:10.808+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:11.633 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:11.633+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:11.635 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:11.636+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:11.850 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:11.851+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:12.657 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:12.658+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:12.678 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:12.679+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:12.810 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:12.811+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:13.695 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:13.696+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:13.727 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:13.728+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:13.847 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:13.848+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:14.707 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:14.708+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:14.743 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:14.744+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:14.820 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:14.821+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:15.673 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:15.672+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:15.770 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:15.769+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:15.862 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:15.862+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:16.701 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:16.701+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:16.756 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:16.756+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:16.849 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:16.849+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:17.727 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:17.726+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:17.743 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:17.743+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:17.844 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:17.844+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:18.755 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:18.755+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:18.777 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:18.776+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:18.888 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:18.888+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:19.719 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:19.719+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:19.734 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:19.734+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:19.868 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:19.868+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:20.686 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:20.686+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:20.764 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:20.763+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:20.844 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:20.844+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:21.721 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:21.721+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:21.784 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:21.784+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:21.883 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:21.882+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:22.702 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:22.701+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:22.824 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:22.824+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:22.874 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:22.874+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:23.655 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:23.655+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:23.819 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:23.819+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:23.901 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:23.900+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:24.669 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:24.668+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:24.796 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:24.795+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:24.919 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:24.919+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:25.653 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:25.653+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:25.821 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:25.821+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:25.957 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:25.956+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:26.678 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:26.678+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:26.814 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:26.814+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:26.953 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:26.952+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:27.713 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:27.713+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:27.851 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:27.851+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:27.934 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:27.934+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:28.715 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:28.715+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:28.898 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:28.898+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:28.899 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:28.898+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:29.719 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:29.719+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:29.891 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:29.891+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:29.895 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:29.895+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:30.707 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:30.707+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:30.852 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:30.852+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:30.903 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:30.903+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:31.745 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:31.745+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:31.825 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:31.825+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:31.897 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:31.897+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:32.703 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:32.703+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:32.788 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:32.787+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:32.861 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:32.860+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:33.699 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:33.698+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:33.791 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:33.791+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:33.874 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:33.874+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:34.677 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:34.677+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:34.827 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:34.827+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:34.884 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:34.884+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:35.699 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:35.699+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:35.856 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:35.855+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:35.920 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:35.919+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:36.667 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:36.667+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:36.834 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:36.834+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:36.942 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:36.942+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:37.691 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:37.691+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:37.849 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:37.848+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:37.923 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:37.923+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:38.706 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:38.706+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:38.899 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:38.898+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:38.937 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:38.937+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:39.747 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:39.747+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:39.884 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:39.884+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:39.982 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:39.981+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:40.782 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:40.781+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:40.922 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:40.922+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:40.955 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:40.955+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:41.769 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:41.769+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:41.899 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:41.899+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:41.938 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:41.937+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:42.780 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:42.780+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:42.881 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:42.881+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:42.911 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:42.911+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:43.753 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:43.752+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:43.844 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:43.844+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:43.899 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:43.898+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:44.745 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:44.745+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:44.894 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:44.894+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:44.940 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:44.940+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:45.781 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:45.780+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:45.930 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:45.930+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:45.969 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:45.968+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:46.801 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:46.800+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:46.910 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:46.911+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:46.959 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:46.960+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:47.760 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:47.760+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:47.902 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:47.902+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:47.914 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:47.914+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:48.801 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:48.802+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:48.866 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:48.867+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:48.893 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:48.893+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:49.767 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:49.767+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:49.864 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:49.865+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:49.892 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:49.893+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:50.810 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:50.811+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:50.859 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:50.860+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:50.912 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:50.913+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:51.763 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:51.763+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:51.830 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:51.831+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:51.912 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:51.913+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:52.767 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:52.767+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:52.811 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:52.812+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:52.864 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:52.865+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:53.820 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:53.818+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:53.859 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:53.860+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:53.873 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:53.874+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:54.779 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:54.779+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:54.880 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:54.881+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:54.897 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:54.897+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:55.801 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:55.802+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:55.840 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:55.841+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:55.882 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:55.883+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:56.756 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:56.756+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:56.804 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:56.804+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:56.899 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:56.900+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:57.758 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:57.759+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:57.780 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:57.781+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:57.898 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:57.899+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:58.737 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:58.737+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:58.821 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:58.822+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:58.850 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:58.850+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:59.732 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:35:59.733+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:59.824 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:35:59.825+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:35:59.827 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:35:59.828+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:00.724 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:00.724+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:00.778 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:00.777+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:00.828 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:00.827+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:01.748 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:01.748+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:01.804 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:01.803+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:01.844 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:01.844+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:02.777 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:02.777+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:02.778 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:02.777+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:02.858 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:02.857+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:03.765 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:03.764+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:03.793 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:03.792+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:03.833 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:03.833+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:04.766 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:04.766+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:04.780 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:04.780+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:04.841 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:04.841+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:05.759 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:05.759+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:05.769 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:05.768+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:05.796 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:05.796+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:06.746 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:06.746+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:06.784 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:06.783+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:06.794 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:06.793+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:07.730 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:07.729+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:07.765 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:07.764+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:07.836 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:07.835+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:08.716 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:08.716+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:08.783 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:08.783+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:08.878 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:08.877+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:09.682 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:09.682+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:09.809 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:09.808+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:09.884 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:09.883+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:10.646 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:10.646+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:10.803 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:10.803+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:10.919 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:10.918+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:11.653 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:11.649+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:11.808 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:11.807+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:11.876 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:11.875+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:12.608 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:12.608+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:12.823 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:12.823+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:12.895 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:12.895+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:13.652 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:13.652+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:13.812 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:13.812+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:13.861 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:13.861+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:14.680 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:14.679+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:14.816 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:14.816+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:14.897 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:14.897+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:15.650 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:15.650+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:15.806 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:15.805+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:15.907 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:15.907+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:16.639 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:16.638+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:16.853 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:16.853+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:16.954 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:16.954+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:17.612 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:17.612+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:17.899 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:17.899+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:17.994 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:17.994+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:18.597 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:18.597+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:18.909 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:18.908+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:18.986 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:18.986+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:19.556 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:19.556+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:19.916 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:19.916+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:19.970 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:19.970+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:20.599 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:20.599+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:20.888 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:20.888+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:20.969 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:20.968+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:21.585 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:21.584+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:21.868 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:21.867+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:21.959 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:21.959+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:22.543 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:22.542+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:22.891 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:22.891+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:22.986 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:22.985+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:23.560 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:23.560+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:23.926 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:23.926+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:23.950 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:23.950+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:24.518 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:24.518+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:24.889 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:24.888+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:24.998 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:24.998+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:25.514 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:25.513+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:25.909 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:25.909+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:25.952 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:25.952+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:26.503 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:26.503+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:26.884 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:26.884+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:26.987 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:26.987+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:27.524 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:27.524+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:27.863 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:27.862+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:28.030 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:28.030+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:28.521 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:28.521+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:28.913 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:28.912+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:29.030 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:29.030+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:29.547 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:29.547+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:29.873 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:29.874+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:30.017 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:30.017+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:30.589 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:30.588+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:30.883 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:30.883+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:30.990 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:30.991+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:31.590 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:31.591+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:31.853 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:31.854+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:31.997 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:31.998+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:32.607 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:32.608+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:32.899 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:32.900+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:33.029 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:33.030+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:33.562 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:33.563+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:33.907 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:33.908+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:34.053 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:34.054+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:34.514 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:34.515+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:34.885 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:34.886+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:35.051 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:35.051+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:35.564 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:35.565+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:35.928 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:35.928+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:36.028 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:36.029+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:36.544 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:36.544+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:36.952 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:36.953+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:37.017 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:37.017+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:37.518 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:37.518+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:37.971 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:37.972+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:38.047 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:38.048+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:38.524 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:38.525+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:38.924 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:38.924+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:39.025 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:39.026+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:39.521 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:39.522+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:39.920 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:39.920+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:39.990 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:39.991+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:40.489 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:40.489+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:40.932 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:40.933+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:40.999 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:41.000+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:41.468 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:41.468+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:41.958 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:41.959+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:41.970 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:41.970+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:42.446 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:42.446+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:42.926 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:42.927+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:42.988 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:42.989+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:43.469 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:43.469+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:43.946 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:43.947+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:44.016 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:44.017+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:44.421 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:44.422+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:44.903 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:44.903+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:44.986 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:44.986+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:45.401 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:45.400+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:45.923 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:45.922+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:45.994 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:45.993+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:46.448 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:46.447+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:46.914 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:46.913+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:47.031 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:47.031+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:47.425 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:47.425+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:47.947 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:47.947+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:48.024 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:48.023+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:48.422 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:48.422+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:48.991 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:48.990+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:49.021 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:49.021+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:49.405 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:49.405+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:49.997 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:49.996+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:50.059 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:50.058+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:50.447 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:50.446+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:51.025 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:51.025+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:51.075 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:51.074+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:51.474 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:51.473+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:52.030 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:52.030+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:52.071 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:52.071+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:52.472 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:52.472+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:52.997 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:52.997+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:53.081 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:53.080+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:53.502 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:53.501+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:54.027 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:54.026+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:54.064 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:54.063+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:54.551 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:54.551+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:55.031 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:55.030+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:55.044 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:55.044+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:55.563 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:55.563+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:56.024 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:56.024+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:56.055 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:56.054+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:56.546 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:56.546+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:57.020 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:57.020+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:57.025 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:57.024+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:57.527 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:57.526+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:57.999 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:57.999+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:58.071 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:58.071+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:58.536 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:58.535+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:59.020 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:36:59.019+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:59.022 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:36:59.022+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:36:59.565 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:36:59.565+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:00.009 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:00.008+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:00.032 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:00.031+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:00.606 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:00.605+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:00.987 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:00.986+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:01.059 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:01.059+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:01.600 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:01.600+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:02.036 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:02.036+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:02.057 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:02.057+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:02.606 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:02.605+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:03.038 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:03.037+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:03.080 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:03.079+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:03.575 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:03.575+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:03.605 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T18:37:03.605+0000 7fccf53da700 0 -- 172.21.15.87:0/3193322219 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7fccd803a500 msgr2=0x7fccd804bd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-24T18:37:04.043 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:04.043+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:04.071 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:04.071+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:04.537 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:04.537+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:05.060 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:05.060+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:05.089 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:05.089+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:05.543 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:05.543+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:06.077 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:06.077+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:06.082 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:06.082+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:06.533 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:06.533+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:07.034 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:07.034+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:07.039 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:07.039+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:07.539 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:07.539+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:08.019 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:08.019+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:08.066 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:08.065+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:08.537 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:08.536+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:09.001 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:09.001+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:09.077 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:09.076+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:09.555 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:09.554+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:10.002 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:10.002+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:10.037 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:10.036+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:10.587 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:10.586+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:10.968 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:10.967+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:11.082 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:11.082+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:11.541 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:11.540+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:11.991 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:11.991+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:12.085 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:12.084+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:12.505 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:12.505+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:12.951 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:12.950+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:13.072 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:13.073+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:13.546 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:13.547+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:13.976 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:13.976+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:14.031 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:14.032+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:14.531 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:14.531+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:15.027 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:15.026+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:15.055 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:15.055+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:15.530 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:15.529+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:16.063 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:16.064+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:16.069 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:16.069+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:16.508 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:16.508+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:17.057 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:17.058+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:17.100 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:17.101+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:17.528 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:17.528+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:18.044 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:18.045+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:18.122 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:18.123+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:18.549 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:18.548+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:19.004 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:19.004+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:19.161 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:19.161+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:19.592 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:19.593+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:20.024 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:20.025+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:20.111 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:20.112+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:20.566 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:20.567+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:21.064 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:21.064+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:21.100 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:21.101+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:21.520 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:21.520+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:22.037 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:22.038+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:22.072 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:22.073+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:22.524 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:22.525+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:23.022 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:23.023+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:23.059 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:23.059+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:23.545 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:23.546+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:24.012 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:24.012+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:24.072 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:24.072+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:24.594 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:24.595+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:25.021 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:25.022+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:25.100 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:25.100+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:25.554 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:25.555+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:25.996 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:25.997+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:26.082 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:26.083+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:26.515 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:26.515+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:26.980 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:26.980+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:27.043 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:27.044+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:27.512 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:27.512+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:27.998 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:27.998+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:28.034 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:28.035+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:28.514 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:28.515+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:29.009 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:29.008+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:29.024 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:29.024+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:29.496 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:29.496+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:29.997 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:29.997+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:30.009 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:30.009+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:30.487 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:30.482+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:31.043 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:31.043+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:31.048 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:31.048+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:31.473 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:31.472+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:32.021 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:32.020+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:32.072 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:32.072+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:32.442 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:32.441+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:32.999 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:32.999+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:33.036 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:33.036+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:33.474 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:33.474+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:34.035 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:34.035+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:34.059 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:34.058+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:34.440 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:34.439+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:35.040 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:35.040+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:35.065 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:35.064+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:35.451 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:35.451+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:36.017 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:36.015+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:36.093 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:36.092+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:36.424 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:36.424+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:36.972 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:36.972+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:37.122 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:37.122+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:37.397 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:37.396+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:37.967 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:37.967+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:38.161 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:38.160+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:38.439 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:38.439+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:39.013 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:39.012+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:39.159 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:39.158+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:39.476 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:39.476+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:39.972 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:39.971+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:40.127 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:40.127+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:40.482 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:40.481+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:40.998 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:40.998+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:41.122 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:41.121+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:41.457 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:41.457+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:41.964 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:41.964+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:42.114 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:42.113+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:42.502 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:42.501+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:42.975 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:42.975+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:43.093 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:43.093+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:43.460 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:43.459+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:43.938 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:43.938+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:44.123 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:44.123+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:44.427 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:44.427+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:44.975 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:44.975+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:45.142 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:45.142+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:45.476 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:45.476+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:45.960 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:45.960+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:46.191 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:46.190+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:46.504 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:46.503+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:46.986 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:46.985+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:47.182 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:47.181+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:47.482 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:47.482+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:47.994 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:47.994+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:48.196 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:48.196+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:48.505 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:48.505+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:49.043 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:49.042+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:49.197 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:49.196+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:49.464 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:49.463+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:50.026 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:50.026+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:50.202 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:50.202+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:50.498 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:50.497+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:51.067 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:51.066+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:51.164 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:51.164+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:51.536 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:51.536+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:52.094 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:52.094+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:52.130 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:52.129+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:52.567 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:52.567+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:53.070 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:53.070+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:53.102 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:53.101+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:53.576 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:53.576+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:54.120 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:54.120+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:54.151 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:54.151+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:54.624 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:54.623+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:55.155 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:55.154+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:55.156 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:55.156+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:55.669 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:55.667+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:56.118 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:56.117+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:56.154 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:56.154+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:56.641 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:56.640+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:57.128 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:57.127+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:57.202 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:57.201+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:57.641 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:57.641+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:58.137 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:58.136+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:58.156 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:58.157+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:58.658 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:58.658+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:59.130 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:37:59.129+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:59.183 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:37:59.184+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:37:59.685 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:37:59.686+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:00.109 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:00.109+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:00.167 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:00.168+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:00.656 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:00.657+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:01.138 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:01.139+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:01.182 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:01.183+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:01.674 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:01.674+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:02.115 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:02.116+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:02.163 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:02.164+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:02.632 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:02.633+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:03.111 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:03.112+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:03.122 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:03.122+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:03.590 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:03.591+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:04.094 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:04.094+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:04.121 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:04.121+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:04.578 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:04.579+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:05.140 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:05.141+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:05.142 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:05.143+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:05.623 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:05.624+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:06.148 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:06.148+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:06.155 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:06.156+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:06.587 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:06.588+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:07.117 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:07.118+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:07.128 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:07.128+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:07.589 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:07.590+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:08.140 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:08.141+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:08.143 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:08.144+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:08.631 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:08.631+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:09.121 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:09.122+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:09.181 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:09.182+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:09.582 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:09.583+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:10.084 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:10.084+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:10.149 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:10.150+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:10.546 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:10.547+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:11.095 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:11.096+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:11.140 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:11.140+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:11.572 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:11.573+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:12.058 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:12.059+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:12.127 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:12.127+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:12.580 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:12.581+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:13.038 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:13.038+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:13.154 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:13.153+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:13.592 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:13.592+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:14.052 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:14.051+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:14.147 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:14.147+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:14.554 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:14.553+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:15.094 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:15.094+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:15.153 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:15.150+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:15.570 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:15.570+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:16.130 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:16.130+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:16.191 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:16.190+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:16.540 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:16.540+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:17.126 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:17.125+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:17.163 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:17.162+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:17.508 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:17.508+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:18.102 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:18.101+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:18.121 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:18.120+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:18.498 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:18.498+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:19.091 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:19.090+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:19.155 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:19.155+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:19.485 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:19.484+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:20.109 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:20.109+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:20.173 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:20.173+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:20.436 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:20.436+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:21.137 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:21.136+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:21.205 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:21.205+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:21.466 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:21.466+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:22.124 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:22.124+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:22.189 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:22.189+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:22.480 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:22.480+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:23.142 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:23.142+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:23.173 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:23.173+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:23.530 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:23.530+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:24.133 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:24.132+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:24.196 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:24.196+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:24.556 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:24.555+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:25.153 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:25.153+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:25.174 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:25.174+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:25.585 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:25.584+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:26.162 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:26.162+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:26.183 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:26.182+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:26.593 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:26.592+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:27.154 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:27.154+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:27.182 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:27.182+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:27.550 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:27.550+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:28.154 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:28.154+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:28.193 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:28.193+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:28.591 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:28.591+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:29.144 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:29.144+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:29.187 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:29.187+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:29.610 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:29.609+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:30.113 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:30.112+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:30.145 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:30.141+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:30.652 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:30.652+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:31.089 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:31.089+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:31.104 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:31.103+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:31.681 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:31.681+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:32.086 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:32.086+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:32.100 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:32.100+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:32.664 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:32.663+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:33.070 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:33.069+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:33.123 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:33.122+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:33.653 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:33.652+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:34.055 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:34.054+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:34.089 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:34.088+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:34.664 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:34.664+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:35.050 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:35.045+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:35.077 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:35.076+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:35.621 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:35.621+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:36.049 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:36.049+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:36.098 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:36.098+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:36.632 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:36.632+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:37.053 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:37.053+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:37.059 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:37.058+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:37.620 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:37.619+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:38.016 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:38.015+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:38.109 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:38.109+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:38.587 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:38.586+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:39.008 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:39.008+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:39.107 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:39.107+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:39.604 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:39.604+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:40.014 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:40.014+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:40.068 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:40.068+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:40.617 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:40.616+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:41.047 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:41.047+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:41.052 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:41.052+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:41.595 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:41.595+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:42.005 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:42.005+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:42.035 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:42.035+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:42.609 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:42.608+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:42.964 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:42.963+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:43.056 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:43.056+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:43.647 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:43.647+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:43.915 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:43.915+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:44.104 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:44.104+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:44.625 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:44.625+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:44.935 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:44.936+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:45.070 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:45.070+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:45.612 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:45.613+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:45.960 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:45.961+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:46.049 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:46.050+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:46.603 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:46.604+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:46.949 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:46.949+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:47.026 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:47.026+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:47.572 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:47.572+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:47.907 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:47.908+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:48.060 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:48.060+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:48.596 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:48.597+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:48.909 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:48.909+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:49.056 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:49.056+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:49.626 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:49.626+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:49.939 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:49.940+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:50.020 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:50.020+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:50.644 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:50.644+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:50.905 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:50.906+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:51.043 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:51.043+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:51.597 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:51.598+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:51.864 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:51.864+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:52.040 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:52.041+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:52.618 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:52.618+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:52.839 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:52.840+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:53.069 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:53.070+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:53.623 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:53.623+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:53.867 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:53.868+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:54.071 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:54.072+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:54.646 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:54.646+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:54.828 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:54.828+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:55.105 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:55.105+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:55.651 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:55.651+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:55.875 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:55.876+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:56.136 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:56.137+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:56.663 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:56.664+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:56.860 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:56.861+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:57.121 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:57.120+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:57.653 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:57.653+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:57.821 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:57.821+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:58.142 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:58.142+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:58.688 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:58.687+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:58.837 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:58.837+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:59.183 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:38:59.182+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:59.686 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:38:59.686+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:38:59.831 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:38:59.831+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:00.224 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:00.223+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:00.730 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:00.729+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:00.851 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:00.850+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:01.240 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:01.240+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:01.721 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:01.721+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:01.825 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:01.825+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:02.263 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:02.263+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:02.746 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:02.746+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:02.794 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:02.793+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:03.285 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:03.284+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:03.756 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:03.755+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:03.833 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:03.833+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:04.273 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:04.272+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:04.775 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:04.774+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:04.882 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:04.881+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:05.233 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:05.233+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:05.728 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:05.727+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:05.851 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:05.850+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:06.278 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:06.277+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:06.696 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:06.696+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:06.869 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:06.868+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:07.304 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:07.304+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:07.650 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:07.650+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:07.849 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:07.849+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:08.284 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:08.284+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:08.693 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:08.693+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:08.896 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:08.895+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:09.301 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:09.301+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:09.669 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:09.669+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:09.944 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:09.943+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:10.336 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:10.336+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:10.693 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:10.693+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:10.951 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:10.951+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:11.326 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:11.325+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:11.667 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:11.666+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:11.960 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:11.960+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:12.309 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:12.309+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:12.710 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:12.710+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:12.919 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:12.919+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:13.264 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:13.264+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:13.705 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:13.705+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:13.872 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:13.872+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:14.219 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:14.219+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:14.715 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:14.715+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:14.890 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:14.889+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:15.228 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:15.227+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:15.704 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:15.703+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:15.855 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:15.854+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:16.215 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:16.214+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:16.718 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:16.717+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:16.903 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:16.903+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:17.173 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:17.173+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:17.761 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:17.761+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:17.863 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:17.863+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:18.173 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:18.173+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:18.754 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:18.753+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:18.828 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:18.827+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:19.172 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:19.171+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:19.770 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:19.769+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:19.813 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:19.813+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:20.136 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:20.136+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:20.783 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:20.782+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:20.827 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:20.827+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:21.157 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:21.156+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:21.779 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:21.778+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:21.784 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:21.784+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:22.205 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:22.204+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:22.824 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:22.823+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:22.829 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:22.828+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:23.165 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:23.164+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:23.849 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:23.848+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:23.874 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:23.873+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:24.140 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:24.140+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:24.804 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:24.804+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:24.843 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:24.842+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:25.186 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:25.186+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:25.815 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:25.815+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:25.842 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:25.841+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:26.196 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:26.197+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:26.799 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:26.798+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:26.856 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:26.855+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:27.159 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:27.158+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:27.812 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:27.813+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:27.850 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:27.850+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:28.191 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:28.191+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:28.815 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:28.815+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:28.851 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:28.851+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:29.157 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:29.158+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:29.800 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:29.800+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:29.802 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:29.803+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:30.198 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:30.199+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:30.769 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:30.770+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:30.844 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:30.844+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:31.163 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:31.163+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:31.809 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:31.809+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:31.846 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:31.846+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:32.142 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:32.142+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:32.767 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:32.767+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:32.866 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:32.867+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:33.176 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:33.177+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:33.816 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:33.816+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:33.833 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:33.833+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:34.173 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:34.174+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:34.853 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:34.854+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:34.855 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:34.856+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:35.146 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:35.147+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:35.840 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:35.840+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:35.856 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:35.857+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:36.162 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:36.163+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:36.865 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:36.865+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:36.865 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:36.865+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:37.161 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:37.161+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:37.840 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:37.840+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:37.886 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:37.886+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:38.208 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:38.208+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:38.811 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:38.811+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:38.917 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:38.917+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:39.213 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:39.213+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:39.851 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:39.851+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:39.889 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:39.889+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:40.226 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:40.226+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:40.890 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:40.885+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:40.932 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:40.931+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:41.227 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:41.226+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:41.847 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:41.846+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:41.943 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:41.943+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:42.237 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:42.237+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:42.893 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:42.892+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:42.980 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:42.980+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:43.234 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:43.233+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:43.927 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:43.926+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:44.029 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:44.029+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:44.261 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:44.261+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:44.886 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:44.885+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:45.016 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:45.016+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:45.310 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:45.309+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:45.921 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:45.921+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:46.061 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:46.060+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:46.302 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:46.302+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:46.953 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:46.952+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:47.085 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:47.085+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:47.303 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:47.303+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:47.961 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:47.960+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:48.091 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:48.090+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:48.305 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:48.304+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:48.940 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:48.940+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:49.118 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:49.118+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:49.342 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:49.342+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:49.904 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:49.904+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:50.140 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:50.140+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:50.300 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:50.300+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:50.938 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:50.937+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:51.128 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:51.128+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:51.285 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:51.284+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:51.950 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:51.950+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:52.132 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:52.131+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:52.334 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:52.333+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:52.989 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:52.989+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:53.177 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:53.177+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:53.372 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:53.371+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:54.008 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:54.008+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:54.188 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:54.187+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:54.362 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:54.361+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:55.015 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:55.015+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:55.220 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:55.220+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:55.399 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:55.398+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:56.065 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:56.064+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:56.216 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:56.215+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:56.382 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:56.381+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:57.062 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:57.062+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:57.215 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:57.215+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:57.364 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:57.363+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:58.102 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:58.101+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:58.171 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:58.170+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:58.369 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:58.368+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:59.068 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:39:59.068+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:59.155 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:39:59.154+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:39:59.356 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:39:59.355+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:00.096 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:00.096+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:00.107 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:00.107+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:00.396 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:00.395+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:01.066 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:01.065+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:01.121 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:01.120+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:01.375 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:01.374+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:02.060 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:02.060+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:02.092 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:02.092+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:02.405 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:02.404+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:03.067 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:03.066+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:03.110 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:03.110+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:03.377 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:03.376+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:04.048 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:04.048+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:04.066 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:04.065+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:04.375 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:04.375+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:05.032 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:05.032+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:05.077 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:05.076+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:05.397 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:05.397+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:06.046 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:06.045+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:06.059 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:06.058+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:06.354 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:06.354+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:07.042 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:07.042+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:07.089 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:07.088+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:07.399 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:07.399+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:08.061 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:08.060+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:08.092 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:08.092+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:08.440 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:08.439+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:09.042 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:09.042+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:09.086 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:09.086+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:09.469 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:09.469+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:10.056 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:10.055+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:10.098 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:10.098+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:10.507 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:10.506+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:11.046 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:11.045+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:11.070 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:11.070+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:11.515 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:11.515+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:12.042 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:12.042+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:12.103 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:12.104+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:12.507 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:12.507+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:13.057 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:13.057+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:13.069 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:13.069+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:13.501 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:13.501+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:14.070 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:14.070+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:14.077 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:14.078+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:14.534 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:14.534+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:15.116 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:15.116+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:15.126 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:15.127+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:15.548 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:15.548+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:16.089 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:16.090+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:16.101 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:16.101+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:16.576 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:16.577+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:17.073 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:17.073+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:17.077 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:17.078+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:17.541 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:17.541+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:18.035 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:18.036+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:18.038 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:18.039+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:18.561 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:18.561+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:19.061 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:19.062+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:19.069 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:19.070+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:19.586 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:19.586+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:20.055 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:20.056+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:20.107 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:20.107+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:20.547 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:20.547+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:21.037 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:21.038+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:21.065 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:21.065+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:21.565 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:21.565+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:22.020 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:22.020+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:22.073 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:22.073+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:22.542 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:22.543+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:22.990 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:22.990+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:23.077 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:23.077+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:23.542 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:23.543+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:23.986 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:23.987+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:24.087 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:24.088+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:24.509 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:24.509+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:25.031 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:25.030+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:25.090 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:25.089+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:25.488 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:25.488+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:26.013 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:26.013+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:26.131 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:26.131+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:26.483 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:26.482+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:27.064 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:27.063+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:27.163 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:27.163+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:27.479 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:27.479+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:28.105 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:28.105+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:28.120 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:28.119+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:28.511 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:28.510+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:29.114 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:29.113+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:29.127 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:29.126+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:29.463 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:29.462+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:30.136 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:30.136+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:30.177 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:30.176+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:30.506 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:30.505+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:31.102 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:31.101+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:31.208 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:31.208+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:31.524 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:31.523+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:32.100 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:32.100+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:32.250 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:32.249+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:32.497 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:32.497+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:33.105 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:33.105+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:33.205 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:33.205+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:33.467 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:33.467+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:34.109 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:34.109+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:34.163 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:34.163+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:34.475 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:34.475+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:35.155 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:35.155+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:35.213 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:35.213+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:35.430 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:35.430+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:36.159 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:36.159+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:36.200 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:36.200+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:36.430 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:36.429+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:37.167 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:37.167+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:37.205 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:37.204+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:37.466 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:37.465+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:38.166 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:38.165+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:38.166 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:38.165+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:38.492 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:38.491+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:39.138 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:39.134+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:39.186 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:39.186+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:39.503 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:39.502+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:40.162 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:40.162+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:40.214 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:40.213+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:40.495 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:40.494+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:41.125 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:41.125+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:41.166 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:41.165+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:41.538 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:41.537+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:42.088 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:42.087+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:42.197 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:42.197+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:42.555 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:42.554+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:43.091 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:43.090+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:43.167 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:43.167+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:43.582 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:43.581+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:44.120 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:44.120+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:44.190 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:44.189+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:44.572 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:44.572+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:45.148 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:45.147+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:45.181 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:45.180+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:45.580 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:45.580+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:46.144 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:46.143+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:46.155 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:46.155+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:46.566 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:46.565+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:47.161 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:47.161+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:47.182 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:47.181+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:47.576 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:47.576+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:48.131 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:48.131+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:48.215 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:48.214+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:48.605 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:48.605+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:49.165 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:49.164+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:49.208 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:49.207+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:49.627 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:49.627+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:50.152 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:50.151+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:50.239 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:50.238+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:50.617 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:50.616+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:51.135 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:51.135+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:51.209 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:51.209+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:51.635 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:51.635+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:52.116 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:52.115+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:52.237 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:52.236+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:52.656 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:52.655+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:53.070 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:53.069+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:53.251 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:53.250+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:53.624 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:53.624+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:54.104 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:54.103+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:54.230 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:54.230+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:54.596 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:54.596+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:55.074 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:55.074+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:55.224 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:55.224+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:55.601 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:55.601+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:56.040 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:56.040+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:56.237 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:56.237+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:56.648 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:56.648+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:57.065 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:57.065+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:57.218 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:57.219+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:57.665 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:57.666+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:58.019 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:58.020+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:58.181 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:58.181+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:58.699 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:58.700+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:59.059 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:40:59.059+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:59.198 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:40:59.199+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:40:59.718 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:40:59.719+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:00.032 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:00.032+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:00.208 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:00.209+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:00.759 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:00.760+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:00.988 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:00.988+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:01.160 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:01.161+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:01.774 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:01.774+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:01.939 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:01.940+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:02.207 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:02.208+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:02.802 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:02.802+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:02.903 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:02.903+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:03.221 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:03.221+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:03.787 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:03.788+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:03.898 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:03.898+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:04.202 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:04.203+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:04.757 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:04.757+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:04.858 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:04.858+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:05.163 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:05.163+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:05.771 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:05.772+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:05.900 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:05.901+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:06.149 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:06.150+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:06.754 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:06.754+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:06.945 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:06.945+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:07.119 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:07.119+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:07.787 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:07.788+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:07.904 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:07.904+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:08.168 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:08.168+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:08.754 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:08.753+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:08.908 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:08.907+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:09.181 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:09.181+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:09.706 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:09.706+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:09.919 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:09.919+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:10.179 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:10.179+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:10.730 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:10.730+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:10.965 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:10.964+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:11.197 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:11.197+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:11.770 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:11.769+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:11.941 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:11.941+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:12.185 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:12.184+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:12.786 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:12.786+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:12.979 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:12.979+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:13.141 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:13.141+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:13.773 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:13.773+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:14.029 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:14.028+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:14.109 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:14.108+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:14.789 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:14.788+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:15.013 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:15.013+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:15.077 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:15.076+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:15.832 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:15.831+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:16.059 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:16.059+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:16.112 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:16.112+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:16.846 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:16.845+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:17.109 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:17.107+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:17.152 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:17.151+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:17.824 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:17.823+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:18.097 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:18.097+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:18.125 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:18.125+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:18.803 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:18.802+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:19.083 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:19.082+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:19.132 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:19.131+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:19.800 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:19.800+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:20.119 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:20.119+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:20.147 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:20.146+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:20.779 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:20.778+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:21.107 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:21.106+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:21.168 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:21.167+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:21.783 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:21.782+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:22.091 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:22.091+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:22.216 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:22.215+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:22.805 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:22.804+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:23.124 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:23.123+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:23.246 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:23.245+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:23.827 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:23.827+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:24.162 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:24.162+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:24.213 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:24.213+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:24.848 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:24.848+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:25.160 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:25.160+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:25.173 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:25.172+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:25.802 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:25.801+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:26.134 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:26.134+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:26.150 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:26.149+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:26.777 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:26.776+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:27.115 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:27.114+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:27.127 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:27.126+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:27.787 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:27.787+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:28.101 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:28.101+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:28.156 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:28.156+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:28.821 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:28.821+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:29.132 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:29.131+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:29.150 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:29.149+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:29.813 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:29.812+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:30.138 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:30.137+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:30.157 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:30.156+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:30.844 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:30.843+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:31.127 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:31.127+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:31.169 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:31.169+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:31.893 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:31.892+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:32.133 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:32.133+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:32.148 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:32.148+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:32.891 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:32.890+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:33.127 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:33.126+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:33.187 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:33.187+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:33.938 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:33.937+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:34.119 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:34.118+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:34.145 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:34.145+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:34.977 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:34.976+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:35.106 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:35.106+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:35.177 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:35.176+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:36.017 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:36.016+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:36.078 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:36.077+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:36.187 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:36.186+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:37.034 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:37.033+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:37.118 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:37.117+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:37.214 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:37.213+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:38.049 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:38.048+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:38.157 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:38.157+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:38.258 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:38.257+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:39.020 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:39.020+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:39.164 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:39.165+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:39.253 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:39.253+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:40.008 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:40.008+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:40.146 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:40.147+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:40.296 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:40.297+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:41.056 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:41.056+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:41.104 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:41.105+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:41.336 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:41.337+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:42.057 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:42.058+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:42.114 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:42.114+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:42.289 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:42.290+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:43.084 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:43.084+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:43.121 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:43.121+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:43.322 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:43.322+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:44.098 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:44.098+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:44.103 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:44.103+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:44.316 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:44.316+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:45.091 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:45.091+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:45.128 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:45.129+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:45.285 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:45.286+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:46.130 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:46.131+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:46.179 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:46.179+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:46.262 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:46.262+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:47.107 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:47.108+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:47.183 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:47.183+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:47.220 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:47.221+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:48.126 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:48.126+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:48.199 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:48.199+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:48.236 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:48.236+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:49.079 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:49.079+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:49.223 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:49.223+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:49.247 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:49.247+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:50.075 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:50.075+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:50.179 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:50.180+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:50.284 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:50.284+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:51.066 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:51.066+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:51.134 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:51.134+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:51.292 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:51.292+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:52.086 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:52.085+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:52.172 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:52.171+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:52.269 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:52.268+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:53.048 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:53.047+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:53.170 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:53.169+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:53.223 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:53.222+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:54.019 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:54.018+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:54.139 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:54.139+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:54.188 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:54.187+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:54.975 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:54.974+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:55.114 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:55.114+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:55.147 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:55.147+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:55.929 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:55.929+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:56.127 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:56.126+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:56.161 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:56.160+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:56.972 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:56.971+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:57.090 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:57.090+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:57.207 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:57.206+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:57.963 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:57.962+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:58.128 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:58.127+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:58.205 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:58.205+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:58.976 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:41:58.976+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:59.086 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:41:59.086+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:41:59.254 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:41:59.253+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:00.014 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:00.013+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:00.107 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:00.106+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:00.257 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:00.257+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:01.058 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:01.058+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:01.059 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:01.059+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:01.273 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:01.272+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:02.041 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:02.040+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:02.054 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:02.053+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:02.240 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:02.240+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:03.031 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:03.030+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:03.099 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:03.099+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:03.287 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:03.287+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:04.046 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:04.046+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:04.113 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:04.113+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:04.304 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:04.304+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:05.091 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:05.090+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:05.147 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:05.147+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:05.331 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:05.330+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:06.084 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:06.083+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:06.183 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:06.182+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:06.315 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:06.314+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:07.120 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:07.119+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:07.227 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:07.227+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:07.292 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:07.292+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:08.102 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:08.102+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:08.250 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:08.250+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:08.282 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:08.281+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:09.141 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:09.141+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:09.228 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:09.227+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:09.234 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:09.234+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:10.158 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:10.158+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:10.203 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:10.203+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:10.261 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:10.260+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:11.202 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:11.202+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:11.235 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:11.234+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:11.274 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:11.274+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:12.191 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:12.190+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:12.249 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:12.249+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:12.318 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:12.318+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:13.185 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:13.184+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:13.210 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:13.210+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:13.295 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:13.295+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:14.139 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:14.138+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:14.175 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:14.174+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:14.255 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:14.254+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:15.162 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:15.161+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:15.218 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:15.217+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:15.248 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:15.247+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:16.187 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:16.186+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:16.200 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:16.199+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:16.250 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:16.249+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:17.171 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:17.170+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:17.210 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:17.209+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:17.227 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:17.227+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:18.158 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:18.158+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:18.244 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:18.243+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:18.274 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:18.274+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:19.200 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:19.200+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:19.212 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:19.212+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:19.267 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:19.266+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:20.188 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:20.188+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:20.205 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:20.204+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:20.265 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:20.264+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:21.166 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:21.165+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:21.212 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:21.212+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:21.276 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:21.276+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:22.186 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:22.185+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:22.207 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:22.207+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:22.295 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:22.295+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:23.163 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:23.163+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:23.188 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:23.189+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:23.336 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:23.337+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:24.158 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:24.158+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:24.176 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:24.176+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:24.322 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:24.323+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:25.118 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:25.118+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:25.215 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:25.215+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:25.291 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:25.291+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:26.117 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:26.118+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:26.242 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:26.243+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:26.279 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:26.280+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:27.166 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:27.167+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:27.256 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:27.256+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:27.324 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:27.324+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:28.154 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:28.155+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:28.213 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:28.213+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:28.322 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:28.323+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:29.193 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:29.193+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:29.245 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:29.246+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:29.312 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:29.312+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:30.226 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:30.227+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:30.244 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:30.244+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:30.303 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:30.304+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:31.232 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:31.233+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:31.276 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:31.277+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:31.304 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:31.305+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:32.254 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:32.254+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:32.267 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:32.268+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:32.294 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:32.294+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:33.273 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:33.273+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:33.277 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:33.277+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:33.324 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:33.325+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:34.246 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:34.247+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:34.270 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:34.270+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:34.301 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:34.301+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:35.268 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:35.267+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:35.290 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:35.289+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:35.344 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:35.343+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:36.243 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:36.242+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:36.323 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:36.322+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:36.357 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:36.356+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:37.261 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:37.260+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:37.329 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:37.328+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:37.365 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:37.364+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:38.254 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:38.253+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:38.339 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:38.338+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:38.344 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:38.344+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:39.249 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:39.248+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:39.329 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:39.328+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:39.360 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:39.359+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:40.237 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:40.237+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:40.356 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:40.355+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:40.375 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:40.374+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:41.229 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:41.228+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:41.324 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:41.324+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:41.328 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:41.327+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:42.262 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:42.261+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:42.325 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:42.325+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:42.366 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:42.365+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:43.241 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:43.240+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:43.366 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:43.365+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:43.367 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:43.367+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:44.196 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:44.195+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:44.332 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:44.331+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:44.364 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:44.364+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:45.220 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:45.220+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:45.372 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:45.372+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:45.399 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:45.398+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:46.188 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:46.188+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:46.341 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:46.340+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:46.433 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:46.433+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:47.216 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:47.215+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:47.302 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:47.301+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:47.396 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:47.395+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:48.247 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:48.246+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:48.346 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:48.345+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:48.390 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:48.389+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:49.247 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:49.247+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:49.388 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:49.388+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:49.417 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:49.416+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:50.244 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:50.244+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:50.389 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:50.388+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:50.450 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:50.449+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:51.226 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:51.226+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:51.405 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:51.402+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:51.454 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:51.453+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:52.269 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:52.269+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:52.385 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:52.384+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:52.484 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:52.484+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:53.274 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:53.273+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:53.370 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:53.369+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:53.461 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:53.460+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:54.296 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:54.295+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:54.394 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:54.394+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:54.499 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:54.499+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:55.257 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:55.256+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:55.388 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:55.387+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:55.450 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:55.449+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:56.213 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:56.212+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:56.340 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:56.339+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:56.466 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:56.465+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:57.227 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:57.227+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:57.357 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:57.357+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:57.511 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:57.511+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:58.271 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:58.270+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:58.354 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:58.354+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:58.558 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:58.557+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:59.296 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:42:59.296+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:59.388 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:42:59.387+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:42:59.533 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:42:59.533+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:00.254 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:00.254+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:00.379 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:00.378+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:00.572 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:00.572+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:01.219 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:01.218+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:01.361 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:01.360+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:01.567 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:01.566+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:02.229 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:02.228+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:02.409 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:02.408+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:02.563 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:02.562+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:03.188 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:03.188+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:03.448 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:03.447+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:03.604 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:03.605+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:04.181 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:04.180+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:04.408 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:04.408+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:04.648 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:04.647+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:05.174 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:05.173+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:05.364 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:05.363+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:05.615 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:05.615+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:06.189 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:06.189+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:06.391 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:06.391+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:06.573 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:06.574+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:07.207 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:07.208+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:07.352 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:07.353+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:07.588 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:07.589+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:08.196 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:08.196+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:08.350 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:08.351+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:08.619 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:08.619+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:09.154 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:09.154+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:09.302 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:09.302+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:09.665 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:09.665+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:10.138 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:10.138+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:10.339 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:10.340+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:10.664 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:10.664+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:11.129 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:11.129+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:11.334 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:11.335+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:11.642 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:11.642+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:12.179 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:12.179+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:12.286 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:12.286+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:12.662 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:12.662+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:13.139 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:13.139+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:13.273 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:13.273+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:13.670 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:13.671+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:14.169 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:14.169+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:14.308 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:14.308+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:14.710 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:14.710+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:15.122 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:15.122+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:15.324 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:15.324+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:15.714 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:15.714+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:16.149 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:16.149+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:16.328 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:16.328+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:16.731 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:16.732+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:17.174 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:17.175+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:17.307 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:17.308+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:17.722 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:17.722+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:18.204 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:18.204+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:18.344 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:18.344+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:18.771 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:18.771+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:19.181 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:19.180+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:19.311 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:19.310+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:19.740 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:19.739+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:20.197 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:20.197+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:20.306 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:20.306+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:20.706 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:20.706+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:21.234 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:21.234+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:21.288 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:21.287+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:21.698 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:21.698+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:22.192 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:22.191+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:22.329 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:22.328+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:22.684 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:22.683+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:23.149 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:23.148+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:23.328 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:23.327+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:23.724 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:23.723+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:24.186 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:24.186+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:24.351 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:24.350+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:24.725 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:24.724+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:25.192 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:25.191+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:25.391 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:25.391+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:25.731 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:25.730+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:26.186 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:26.185+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:26.395 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:26.395+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:26.711 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:26.711+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:27.190 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:27.190+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:27.434 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:27.433+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:27.694 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:27.693+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:28.228 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:28.228+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:28.433 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:28.432+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:28.674 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:28.673+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:29.252 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:29.251+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:29.392 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:29.392+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:29.629 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:29.628+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:30.233 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:30.232+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:30.434 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:30.434+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:30.675 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:30.675+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:31.202 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:31.201+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:31.454 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:31.454+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:31.712 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:31.711+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:32.241 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:32.240+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:32.459 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:32.458+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:32.672 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:32.671+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:33.282 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:33.281+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:33.450 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:33.449+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:33.670 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:33.669+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:34.296 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:34.296+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:34.496 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:34.495+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:34.664 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:34.664+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:35.302 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:35.302+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:35.517 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:35.517+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:35.684 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:35.684+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:36.280 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:36.279+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:36.487 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:36.486+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:36.707 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:36.707+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:37.279 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:37.278+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:37.447 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:37.447+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:37.752 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:37.752+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:38.280 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:38.280+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:38.486 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:38.486+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:38.720 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:38.719+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:39.264 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:39.264+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:39.526 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:39.526+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:39.699 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:39.698+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:40.234 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:40.233+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:40.507 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:40.506+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:40.718 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:40.717+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:41.205 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:41.204+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:41.536 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:41.536+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:41.681 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:41.681+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:42.184 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:42.184+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:42.491 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:42.491+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:42.667 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:42.667+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:43.199 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:43.198+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:43.503 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:43.502+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:43.704 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:43.703+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:44.233 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:44.232+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:44.546 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:44.545+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:44.683 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:44.683+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:45.262 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:45.261+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:45.541 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:45.541+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:45.669 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:45.669+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:46.238 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:46.237+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:46.529 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:46.529+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:46.641 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:46.641+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:47.213 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:47.212+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:47.551 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:47.550+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:47.654 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:47.654+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:48.211 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:48.211+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:48.578 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:48.578+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:48.692 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:48.690+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:49.217 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:49.216+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:49.561 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:49.562+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:49.658 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:49.659+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:50.254 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:50.254+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:50.541 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:50.541+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:50.692 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:50.693+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:51.296 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:51.296+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:51.501 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:51.502+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:51.718 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:51.719+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:52.336 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:52.336+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:52.514 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:52.515+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:52.734 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:52.735+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:53.341 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:53.341+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:53.536 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:53.536+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:53.692 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:53.693+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:54.302 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:54.302+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:54.506 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:54.506+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:54.669 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:54.669+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:55.282 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:55.282+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:55.551 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:55.551+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:55.664 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:55.665+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:56.240 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:56.241+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:56.515 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:56.516+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:56.676 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:56.677+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:57.287 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:57.288+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:57.565 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:57.566+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:57.705 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:57.705+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:58.307 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:58.308+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:58.541 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:58.541+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:58.671 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:58.672+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:59.289 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:43:59.290+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:59.543 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:43:59.543+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:43:59.715 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:43:59.716+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:00.249 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:00.250+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:00.535 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:00.536+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:00.706 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:00.707+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:01.281 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:01.281+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:01.516 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:01.516+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:01.662 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:01.662+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:02.252 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:02.252+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:02.483 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:02.482+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:02.696 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:02.695+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:03.253 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:03.253+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:03.473 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:03.472+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:03.692 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:03.691+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:04.245 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:04.245+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:04.454 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:04.453+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:04.682 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:04.681+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:05.279 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:05.279+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:05.461 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:05.460+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:05.726 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:05.725+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:06.249 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:06.249+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:06.438 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:06.438+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:06.726 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:06.726+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:07.276 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:07.276+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:07.488 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:07.488+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:07.714 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:07.713+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:08.251 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:08.250+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:08.536 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:08.535+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:08.725 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:08.724+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:09.267 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:09.266+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:09.583 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:09.583+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:09.704 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:09.704+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:10.235 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:10.234+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:10.562 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:10.561+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:10.689 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:10.688+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:11.236 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:11.235+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:11.567 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:11.567+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:11.713 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:11.712+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:12.271 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:12.270+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:12.567 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:12.566+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:12.672 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:12.671+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:13.232 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:13.231+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:13.563 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:13.563+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:13.704 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:13.704+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:14.215 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:14.215+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:14.523 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:14.522+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:14.717 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:14.716+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:15.173 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:15.172+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:15.487 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:15.487+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:15.717 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:15.716+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:16.192 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:16.191+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:16.454 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:16.453+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:16.766 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:16.765+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:17.241 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:17.241+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:17.416 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:17.416+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:17.754 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:17.754+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:18.280 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:18.279+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:18.370 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:18.369+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:18.778 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:18.778+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:19.233 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:19.232+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:19.327 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:19.326+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:19.788 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:19.787+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:20.259 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:20.259+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:20.352 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:20.351+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:20.820 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:20.820+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:21.217 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:21.216+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:21.326 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:21.325+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:21.864 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:21.863+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:22.255 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:22.254+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:22.291 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:22.290+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:22.850 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:22.850+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:23.275 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:23.274+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:23.301 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:23.300+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:23.821 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:23.821+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:24.282 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:24.281+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:24.284 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:24.283+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:24.858 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:24.858+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:25.258 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:25.257+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:25.263 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:25.263+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:25.848 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:25.848+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:26.240 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:26.239+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:26.308 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:26.307+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:26.877 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:26.876+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:27.197 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:27.197+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:27.323 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:27.323+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:27.914 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:27.913+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:28.149 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:28.148+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:28.292 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:28.292+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:28.895 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:28.894+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:29.180 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:29.179+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:29.258 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:29.257+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:29.875 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:29.875+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:30.204 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:30.204+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:30.253 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:30.252+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:30.903 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:30.903+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:31.237 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:31.237+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:31.254 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:31.254+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:31.882 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:31.882+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:32.243 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:32.242+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:32.259 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:32.258+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:32.891 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:32.892+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:33.218 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:33.219+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:33.256 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:33.256+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:33.898 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:33.899+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:34.222 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:34.222+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:34.284 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:34.284+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:34.896 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:34.897+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:35.244 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:35.244+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:35.258 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:35.258+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:35.886 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:35.886+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:36.266 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:36.266+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:36.302 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:36.303+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:36.914 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:36.914+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:37.251 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:37.251+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:37.292 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:37.292+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:37.944 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:37.944+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:38.250 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:38.250+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:38.341 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:38.341+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:38.960 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:38.961+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:39.209 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:39.210+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:39.306 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:39.306+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:39.937 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:39.937+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:40.213 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:40.213+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:40.354 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:40.355+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:40.894 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:40.894+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:41.225 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:41.225+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:41.382 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:41.383+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:41.908 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:41.908+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:42.216 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:42.217+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:42.432 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:42.432+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:42.893 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:42.893+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:43.181 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:43.181+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:43.477 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:43.477+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:43.884 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:43.883+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:44.208 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:44.208+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:44.486 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:44.486+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:44.876 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:44.877+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:45.180 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:45.180+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:45.512 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:45.511+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:45.918 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:45.918+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:46.175 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:46.175+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:46.541 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:46.540+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:46.910 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:46.910+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:47.159 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:47.158+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:47.559 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:47.558+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:47.919 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:47.918+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:48.196 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:48.195+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:48.512 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:48.511+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:48.908 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:48.908+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:49.218 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:49.217+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:49.479 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:49.478+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:49.927 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:49.927+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:50.222 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:50.221+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:50.488 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:50.487+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:50.927 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:50.927+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:51.205 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:51.204+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:51.521 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:51.520+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:51.930 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:51.929+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:52.185 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:52.185+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:52.559 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:52.558+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:52.907 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:52.907+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:53.191 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:53.190+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:53.587 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:53.587+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:53.914 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:53.914+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:54.228 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:54.227+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:54.559 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:54.558+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:54.904 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:54.904+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:55.213 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:55.213+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:55.520 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:55.519+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:55.908 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:55.908+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:56.169 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:56.168+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:56.543 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:56.542+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:56.918 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:56.917+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:57.200 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:57.200+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:57.585 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:57.585+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:57.904 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:57.903+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:58.164 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:58.164+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:58.625 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:58.624+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:58.932 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:58.932+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:59.117 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:44:59.116+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:59.650 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:44:59.649+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:44:59.894 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:44:59.894+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:00.126 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:00.125+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:00.610 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:00.609+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:00.893 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:00.892+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:01.136 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:01.135+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:01.649 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:01.648+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:01.854 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:01.854+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:02.124 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:02.123+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:02.625 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:02.624+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:02.904 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:02.903+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:03.109 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:03.108+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:03.583 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:03.583+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:03.924 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:03.923+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:04.063 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:04.062+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:04.582 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:04.582+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:04.926 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:04.925+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:05.080 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:05.079+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:05.577 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:05.576+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:05.883 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:05.882+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:06.129 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:06.128+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:06.605 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:06.604+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:06.927 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:06.927+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:07.100 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:07.099+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:07.610 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:07.610+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:07.879 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:07.878+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:08.142 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:08.142+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:08.610 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:08.610+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:08.904 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:08.903+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:09.123 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:09.122+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:09.591 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:09.590+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:09.928 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:09.927+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:10.154 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:10.154+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:10.604 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:10.604+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:10.880 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:10.880+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:11.191 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:11.190+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:11.632 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:11.631+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:11.874 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:11.874+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:12.155 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:12.154+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:12.629 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:12.629+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:12.894 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:12.894+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:13.136 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:13.136+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:13.665 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:13.665+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:13.899 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:13.898+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:14.142 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:14.142+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:14.653 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:14.652+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:14.940 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:14.941+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:15.183 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:15.183+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:15.619 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:15.618+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:15.942 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:15.941+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:16.158 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:16.157+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:16.668 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:16.669+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:16.990 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:16.991+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:17.135 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:17.135+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:17.715 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:17.715+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:18.011 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:18.012+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:18.095 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:18.095+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:18.677 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:18.677+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:19.061 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:19.062+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:19.087 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:19.087+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:19.650 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:19.650+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:20.034 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:20.035+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:20.070 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:20.071+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:20.636 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:20.637+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:21.030 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:21.030+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:21.060 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:21.061+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:21.599 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:21.600+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:21.987 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:21.987+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:22.083 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:22.083+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:22.620 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:22.620+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:22.975 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:22.976+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:23.115 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:23.116+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:23.597 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:23.597+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:24.017 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:24.018+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:24.139 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:24.140+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:24.622 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:24.623+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:24.981 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:24.982+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:25.186 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:25.186+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:25.667 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:25.668+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:25.976 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:25.976+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:26.173 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:26.174+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:26.702 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:26.703+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:26.959 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:26.959+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:27.204 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:27.205+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:27.751 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:27.751+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:27.987 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:27.987+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:28.183 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:28.183+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:28.782 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:28.783+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:29.000 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:29.001+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:29.154 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:29.153+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:29.734 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:29.734+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:29.985 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:29.984+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:30.204 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:30.203+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:30.754 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:30.754+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:30.964 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:30.964+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:31.189 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:31.188+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:31.745 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:31.744+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:31.926 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:31.926+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:32.188 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:32.188+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:32.742 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:32.742+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:32.944 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:32.944+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:33.215 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:33.214+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:33.702 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:33.702+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:33.992 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:33.991+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:34.187 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:34.187+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:34.687 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:34.686+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:35.041 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:35.041+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:35.194 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:35.194+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:35.688 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:35.687+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:36.053 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:36.053+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:36.187 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:36.186+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:36.639 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:36.639+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:37.057 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:37.057+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:37.154 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:37.153+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:37.621 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:37.620+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:38.070 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:38.069+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:38.116 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:38.116+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:38.598 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:38.597+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:39.107 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:39.107+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:39.122 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:39.122+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:39.558 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:39.557+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:40.116 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:40.116+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:40.117 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:40.116+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:40.584 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:40.583+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:41.110 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:41.109+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:41.161 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:41.161+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:41.590 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:41.589+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:42.080 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:42.080+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:42.176 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:42.175+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:42.569 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:42.568+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:43.044 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:43.043+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:43.202 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:43.202+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:43.567 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:43.567+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:44.083 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:44.082+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:44.167 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:44.167+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:44.579 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:44.578+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:45.110 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:45.109+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:45.118 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:45.118+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:45.572 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:45.572+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:46.104 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:46.104+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:46.152 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:46.152+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:46.598 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:46.597+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:47.085 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:47.084+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:47.196 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:47.196+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:47.623 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:47.622+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:48.089 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:48.089+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:48.218 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:48.217+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:48.636 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:48.636+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:49.092 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:49.091+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:49.248 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:49.247+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:49.675 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:49.675+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:50.093 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:50.092+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:50.230 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:50.229+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:50.710 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:50.710+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:51.057 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:51.057+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:51.255 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:51.255+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:51.750 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:51.750+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:52.019 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:52.018+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:52.302 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:52.301+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:52.783 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:52.782+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:53.064 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:53.064+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:53.255 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:53.255+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:53.831 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:53.830+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:54.065 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:54.064+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:54.225 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:54.225+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:54.852 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:54.851+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:55.098 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:55.098+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:55.215 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:55.215+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:55.852 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:55.852+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:56.074 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:56.074+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:56.201 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:56.200+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:56.824 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:56.824+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:57.036 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:57.036+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:57.212 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:57.212+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:57.839 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:57.839+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:58.056 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:58.056+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:58.257 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:58.257+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:58.809 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:58.808+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:59.105 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:45:59.105+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:59.277 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:45:59.277+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:45:59.811 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:45:59.811+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:00.151 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:00.152+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:00.312 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:00.312+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:00.810 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:00.811+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:01.177 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:01.177+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:01.285 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:01.286+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:01.780 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:01.780+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:02.140 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:02.141+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:02.258 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:02.258+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:02.755 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:02.755+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:03.097 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:03.098+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:03.234 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:03.234+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:03.717 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:03.718+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:04.130 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:04.131+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:04.203 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:04.203+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:04.705 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:04.705+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:05.091 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:05.091+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:05.233 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:05.234+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:05.729 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:05.729+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:06.102 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:06.102+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:06.185 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:06.186+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:06.691 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:06.692+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:07.126 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:07.126+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:07.193 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:07.194+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:07.695 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:07.696+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:08.083 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:08.083+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:08.194 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:08.195+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:08.668 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:08.669+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:09.051 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:09.051+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:09.238 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:09.238+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:09.629 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:09.629+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:10.009 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:10.009+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:10.274 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:10.274+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:10.648 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:10.649+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:10.991 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:10.992+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:11.290 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:11.290+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:11.650 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:11.651+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:11.986 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:11.987+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:12.316 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:12.316+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:12.625 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:12.625+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:13.020 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:13.020+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:13.296 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:13.295+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:13.603 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:13.602+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:13.973 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:13.972+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:14.309 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:14.309+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:14.591 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:14.590+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:14.940 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:14.939+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:15.339 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:15.338+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:15.621 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:15.621+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:15.962 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:15.961+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:16.387 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:16.386+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:16.630 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:16.629+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:16.918 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:16.918+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:17.379 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:17.379+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:17.637 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:17.637+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:17.891 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:17.891+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:18.379 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:18.378+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:18.624 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:18.623+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:18.926 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:18.926+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:19.371 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:19.371+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:19.660 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:19.659+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:19.955 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:19.955+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:20.345 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:20.344+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:20.666 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:20.665+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:20.978 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:20.978+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:21.315 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:21.315+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:21.706 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:21.705+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:21.938 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:21.938+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:22.364 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:22.363+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:22.662 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:22.661+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:22.926 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:22.925+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:23.373 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:23.373+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:23.682 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:23.682+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:23.905 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:23.905+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:24.368 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:24.367+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:24.675 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:24.675+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:24.917 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:24.916+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:25.344 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:25.343+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:25.689 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:25.689+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:25.870 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:25.869+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:26.321 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:26.321+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:26.730 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:26.730+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:26.853 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:26.852+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:27.321 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:27.321+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:27.713 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:27.712+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:27.862 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:27.861+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:28.350 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:28.349+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:28.702 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:28.701+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:28.886 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:28.886+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:29.328 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:29.327+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:29.724 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:29.723+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:29.893 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:29.893+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:30.280 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:30.279+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:30.701 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:30.701+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:30.861 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:30.860+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:31.309 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:31.308+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:31.681 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:31.680+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:31.888 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:31.888+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:32.312 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:32.312+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:32.675 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:32.675+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:32.912 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:32.911+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:33.351 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:33.350+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:33.715 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:33.715+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:33.888 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:33.887+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:34.363 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:34.362+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:34.706 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:34.706+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:34.896 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:34.896+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:35.401 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:35.400+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:35.671 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:35.671+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:35.873 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:35.873+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:36.427 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:36.427+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:36.642 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:36.641+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:36.923 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:36.923+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:37.468 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:37.467+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:37.686 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:37.686+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:37.889 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:37.889+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:38.489 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:38.488+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:38.659 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:38.659+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:38.863 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:38.863+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:39.466 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:39.466+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:39.614 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:39.614+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:39.825 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:39.825+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:40.439 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:40.438+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:40.574 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:40.574+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:40.838 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:40.837+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:41.451 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:41.450+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:41.572 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:41.572+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:41.841 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:41.841+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:42.415 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:42.415+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:42.561 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:42.560+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:42.825 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:42.824+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:43.371 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:43.371+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:43.605 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:43.604+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:43.799 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:43.800+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:44.333 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:44.334+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:44.645 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:44.646+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:44.784 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:44.785+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:45.291 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:45.292+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:45.628 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:45.628+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:45.743 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:45.743+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:46.331 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:46.332+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:46.600 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:46.601+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:46.746 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:46.746+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:47.298 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:47.298+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:47.623 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:47.624+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:47.764 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:47.764+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:48.292 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:48.292+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:48.640 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:48.640+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:48.781 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:48.782+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:49.306 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:49.306+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:49.596 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:49.597+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:49.791 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:49.792+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:50.268 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:50.268+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:50.620 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:50.621+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:50.755 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:50.756+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:51.233 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:51.234+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:51.634 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:51.634+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:51.740 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:51.741+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:52.217 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:52.217+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:52.661 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:52.662+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:52.740 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:52.741+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:53.250 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:53.250+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:53.693 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:53.694+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:53.701 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:53.702+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:54.268 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:54.269+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:54.681 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:54.682+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:54.719 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:54.720+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:55.300 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:55.300+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:55.675 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:55.676+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:55.721 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:55.722+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:56.287 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:56.288+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:56.683 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:56.683+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:56.712 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:56.711+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:57.330 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:57.330+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:57.667 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:57.667+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:57.732 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:57.731+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:58.365 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:58.365+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:58.663 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:58.662+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:58.683 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:58.682+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:59.403 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:46:59.402+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:59.646 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:46:59.646+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:46:59.656 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:46:59.656+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:00.378 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:00.376+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:00.605 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:00.605+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:00.622 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:00.621+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:01.404 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:01.404+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:01.603 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:01.602+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:01.673 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:01.670+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:02.415 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:02.415+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:02.559 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:02.559+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:02.632 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:02.632+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:03.462 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:03.461+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:03.565 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:03.565+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:03.636 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:03.635+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:04.491 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:04.490+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:04.548 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:04.547+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:04.603 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:04.603+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:05.500 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:05.500+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:05.526 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:05.525+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:05.618 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:05.618+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:06.528 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:06.528+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:06.536 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:06.535+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:06.651 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:06.651+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:07.482 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:07.481+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:07.513 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:07.513+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:07.690 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:07.690+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:08.507 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:08.507+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:08.515 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:08.515+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:08.665 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:08.665+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:09.477 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:09.477+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:09.502 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:09.502+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:09.654 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:09.653+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:10.504 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:10.503+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:10.533 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:10.533+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:10.687 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:10.686+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:11.518 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:11.517+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:11.581 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:11.581+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:11.638 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:11.638+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:12.563 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:12.563+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:12.628 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:12.627+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:12.641 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:12.641+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:13.570 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:13.570+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:13.597 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:13.597+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:13.677 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:13.676+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:14.578 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:14.577+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:14.619 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:14.619+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:14.683 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:14.683+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:15.543 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:15.542+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:15.583 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:15.582+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:15.688 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:15.688+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:16.589 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:16.589+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:16.621 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:16.621+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:16.665 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:16.664+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:17.559 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:17.558+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:17.583 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:17.583+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:17.667 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:17.667+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:18.529 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:18.529+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:18.620 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:18.619+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:18.661 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:18.661+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:19.493 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:19.492+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:19.571 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:19.571+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:19.678 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:19.677+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:20.525 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:20.524+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:20.607 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:20.607+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:20.653 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:20.653+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:21.497 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:21.496+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:21.591 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:21.591+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:21.632 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:21.632+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:22.477 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:22.477+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:22.586 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:22.586+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:22.653 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:22.653+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:23.479 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:23.478+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:23.570 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:23.570+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:23.612 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:23.612+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:24.504 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:24.503+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:24.559 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:24.558+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:24.659 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:24.658+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:25.486 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:25.486+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:25.607 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:25.607+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:25.624 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:25.624+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:26.515 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:26.516+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:26.583 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:26.582+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:26.623 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:26.622+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:27.527 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:27.527+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:27.631 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:27.632+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:27.636 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:27.636+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:28.518 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:28.518+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:28.595 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:28.596+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:28.656 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:28.657+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:29.535 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:29.535+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:29.613 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:29.613+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:29.642 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:29.643+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:30.492 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:30.493+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:30.604 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:30.605+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:30.614 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:30.615+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:31.527 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:31.528+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:31.575 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:31.575+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:31.662 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:31.662+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:32.496 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:32.497+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:32.562 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:32.563+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:32.638 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:32.639+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:33.515 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:33.516+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:33.545 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:33.546+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:33.594 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:33.594+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:34.475 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:34.476+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:34.588 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:34.589+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:34.637 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:34.638+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:35.479 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:35.480+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:35.580 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:35.581+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:35.619 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:35.620+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:36.507 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:36.507+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:36.613 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:36.614+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:36.657 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:36.658+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:37.463 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:37.463+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:37.648 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:37.648+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:37.652 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:37.652+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:38.461 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:38.462+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:38.603 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:38.603+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:38.692 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:38.693+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:39.423 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:39.424+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:39.624 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:39.624+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:39.665 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:39.665+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:40.440 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:40.440+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:40.602 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:40.601+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 4 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:40.710 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:40.709+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:41.442 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:41.441+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:41.638 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:41.638+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 3 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:41.746 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:41.745+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:42.492 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:42.491+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:42.685 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:42.685+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 3 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:42.721 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:42.720+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:43.504 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:43.504+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:43.708 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:43.708+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 3 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:43.712 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:43.711+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:44.469 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:44.469+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:44.713 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:44.713+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:44.727 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:44.727+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 3 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:45.489 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:45.489+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:45.688 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:45.687+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 3 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:45.729 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:45.729+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:46.507 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:46.507+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:46.651 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:46.650+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 3 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:46.748 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:46.747+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:47.513 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:47.512+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:47.648 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:47.647+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 3 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:47.791 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:47.790+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:48.529 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:48.529+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:48.656 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:48.655+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 3 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:48.822 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:48.822+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:49.574 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:49.574+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:49.623 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:49.622+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 3 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:49.816 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:49.816+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:50.542 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:50.542+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:50.589 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:50.588+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 3 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:50.858 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:50.858+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:51.558 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:51.558+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 3 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:51.573 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:51.573+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:51.882 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:51.881+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:52.540 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:52.539+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:52.567 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:52.567+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 3 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:52.926 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:52.925+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:53.569 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:53.569+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:53.594 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:53.594+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 3 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:53.921 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:53.921+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:54.587 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:54.586+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 3 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:54.613 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:54.612+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:54.905 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:54.905+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:55.570 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:55.570+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 3 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:55.594 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:55.593+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:55.947 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:55.946+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:56.545 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:56.544+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:56.618 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:56.617+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 3 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:56.900 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:56.900+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:57.559 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:57.559+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:57.596 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:57.596+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 3 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:57.887 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:57.886+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:58.586 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:58.586+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 3 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:58.597 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:58.596+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:58.900 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:58.900+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:59.625 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:47:59.625+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 3 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:59.636 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:47:59.636+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:47:59.950 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:47:59.950+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:00.640 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:00.639+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 3 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:00.641 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:00.641+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:00.920 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:00.919+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:01.642 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:01.642+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 3 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:01.692 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:01.691+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:01.889 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:01.888+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:02.679 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:02.678+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 3 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:02.736 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:02.736+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:02.866 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:02.866+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:03.704 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:03.704+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 3 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:03.778 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:03.777+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:03.907 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:03.907+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:04.730 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:04.730+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 3 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:04.795 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:04.795+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:04.879 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:04.879+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:05.747 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:05.747+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 3 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:05.792 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:05.792+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:05.886 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:05.885+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:06.698 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:06.698+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 3 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:06.819 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:06.818+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:06.896 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:06.895+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:07.746 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:07.746+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 3 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:07.830 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:07.830+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:07.909 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:07.909+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:08.713 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:08.713+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 3 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:08.819 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:08.819+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:08.882 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:08.881+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:09.677 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:09.677+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 3 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:09.834 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:09.834+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:09.857 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:09.857+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:10.647 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:10.647+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 3 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:10.791 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:10.791+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:10.881 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:10.881+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:11.664 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:11.665+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:11.836 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:11.837+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:11.838 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:11.838+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:12.680 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:12.681+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:12.866 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:12.866+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:12.878 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:12.879+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:13.688 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:13.689+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:13.855 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:13.855+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:13.869 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:13.869+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:14.704 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:14.704+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:14.845 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:14.845+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:14.881 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:14.881+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:15.721 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:15.721+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:15.888 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:15.888+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:15.903 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:15.904+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:16.763 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:16.763+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:16.933 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:16.933+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:16.948 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:16.949+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:17.740 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:17.741+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:17.926 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:17.927+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:17.981 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:17.981+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:18.759 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:18.759+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:18.900 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:18.901+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:18.974 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:18.975+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:19.730 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:19.731+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:19.909 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:19.909+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:20.001 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:20.001+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:20.692 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:20.693+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:20.910 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:20.910+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:20.953 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:20.953+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:21.693 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:21.694+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:21.896 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:21.897+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:21.997 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:21.997+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:22.703 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:22.704+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:22.856 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:22.857+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:23.045 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:23.045+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:23.741 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:23.742+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:23.852 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:23.853+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:24.052 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:24.052+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:24.779 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:24.779+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:24.836 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:24.836+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:25.034 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:25.034+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:25.781 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:25.779+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:25.801 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:25.800+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:26.013 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:26.012+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:26.759 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:26.759+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:26.795 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:26.794+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:26.964 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:26.963+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:27.718 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:27.718+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:27.842 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:27.842+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:27.936 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:27.936+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:28.669 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:28.669+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:28.822 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:28.822+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:28.966 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:28.966+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:29.701 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:29.701+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:29.788 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:29.787+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:29.985 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:29.985+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:30.729 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:30.729+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:30.832 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:30.832+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:31.018 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:31.018+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:31.692 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:31.691+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:31.862 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:31.862+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:31.994 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:31.994+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:32.668 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:32.668+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:32.884 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:32.884+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:33.003 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:33.003+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:33.639 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:33.638+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:33.862 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:33.862+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:33.965 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:33.965+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:34.599 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:34.598+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:34.878 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:34.877+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:34.955 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:34.955+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:35.560 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:35.560+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:35.879 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:35.879+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:35.932 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:35.932+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:36.603 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:36.603+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:36.895 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:36.895+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:36.973 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:36.972+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:37.628 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:37.627+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:37.871 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:37.871+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:38.000 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:38.000+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:38.641 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:38.641+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:38.862 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:38.862+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:39.040 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:39.040+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:39.599 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:39.599+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:39.856 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:39.856+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:40.022 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:40.022+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:40.585 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:40.584+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:40.874 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:40.874+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:41.069 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:41.068+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:41.602 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:41.602+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:41.915 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:41.915+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:42.072 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:42.072+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:42.598 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:42.598+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:42.964 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:42.964+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:43.045 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:43.044+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:43.554 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:43.554+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:43.981 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:43.980+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:44.039 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:44.038+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:44.528 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:44.528+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:44.973 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:44.973+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:45.002 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:45.001+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:45.552 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:45.551+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:45.985 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:45.985+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:46.036 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:46.035+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:46.554 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:46.554+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:47.021 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:47.020+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:47.022 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:47.022+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:47.529 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:47.529+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:48.040 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:48.040+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:48.058 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:48.057+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:48.513 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:48.513+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:49.068 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:49.068+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:49.090 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:49.090+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:49.514 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:49.513+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:50.086 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:50.086+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:50.094 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:50.094+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:50.498 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:50.497+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:51.058 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:51.057+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:51.113 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:51.113+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:51.516 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:51.516+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:52.018 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:52.017+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:52.119 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:52.118+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:52.505 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:52.505+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:53.025 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:53.025+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:53.151 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:53.151+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:53.464 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:53.463+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:54.026 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:54.026+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:54.119 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:54.119+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:54.431 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:54.431+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:54.998 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:54.999+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:55.157 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:55.157+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:55.473 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:55.473+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:56.004 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:56.004+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:56.176 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:56.176+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:56.499 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:56.499+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:57.021 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:57.021+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:57.207 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:57.208+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:57.483 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:57.484+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:58.012 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:58.012+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:58.257 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:58.257+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:58.528 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:58.528+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:59.016 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:48:59.017+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:59.271 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:48:59.272+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:48:59.570 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:48:59.570+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:00.061 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:00.061+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:00.281 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:00.280+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:00.557 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:00.558+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:01.101 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:01.102+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:01.302 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:01.302+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:01.552 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:01.553+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:02.152 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:02.152+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:02.343 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:02.344+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:02.551 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:02.551+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:03.166 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:03.166+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:03.374 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:03.374+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:03.551 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:03.551+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:04.206 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:04.207+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:04.327 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:04.328+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:04.540 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:04.540+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:05.180 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:05.181+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:05.372 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:05.370+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:05.564 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:05.565+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:06.170 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:06.171+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:06.393 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:06.394+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:06.592 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:06.593+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:07.168 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:07.168+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:07.353 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:07.353+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:07.556 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:07.556+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:08.163 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:08.162+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:08.341 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:08.340+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:08.512 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:08.512+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:09.179 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:09.179+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:09.359 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:09.359+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:09.473 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:09.473+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:10.189 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:10.189+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:10.345 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:10.345+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:10.503 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:10.503+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:11.151 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:11.151+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:11.304 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:11.303+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:11.491 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:11.490+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:12.142 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:12.141+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:12.350 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:12.349+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:12.453 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:12.453+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:13.133 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:13.133+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:13.348 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:13.347+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:13.434 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:13.434+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:14.180 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:14.179+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:14.360 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:14.359+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:14.388 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:14.388+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:15.205 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:15.205+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:15.349 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:15.349+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:15.350 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:15.350+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:16.163 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:16.163+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:16.323 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:16.323+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:16.387 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:16.387+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:17.132 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:17.132+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:17.351 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:17.351+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:17.364 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:17.363+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:18.128 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:18.128+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:18.327 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:18.326+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:18.329 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:18.328+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:19.095 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:19.094+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:19.320 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:19.320+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:19.359 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:19.359+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:20.123 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:20.123+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:20.294 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:20.293+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:20.385 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:20.385+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:21.096 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:21.096+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:21.249 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:21.248+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:21.339 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:21.339+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:22.083 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:22.083+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:22.295 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:22.294+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:22.318 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:22.318+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:23.125 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:23.125+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:23.275 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:23.274+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:23.364 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:23.364+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:24.162 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:24.162+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:24.229 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:24.229+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:24.372 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:24.372+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:25.185 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:25.185+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:25.195 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:25.195+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:25.397 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:25.397+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:26.148 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:26.148+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:26.215 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:26.215+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:26.432 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:26.431+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:27.155 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:27.155+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:27.260 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:27.260+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:27.481 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:27.481+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:28.123 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:28.123+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:28.251 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:28.251+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:28.437 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:28.437+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:29.151 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:29.151+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:29.266 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:29.266+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:29.456 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:29.456+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:30.197 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:30.196+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:30.235 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:30.235+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:30.440 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:30.439+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:31.201 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:31.201+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:31.243 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:31.242+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:31.431 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:31.430+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:32.194 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:32.193+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:32.226 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:32.226+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:32.431 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:32.430+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:33.187 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:33.187+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:33.220 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:33.219+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:33.429 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:33.429+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:34.233 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:34.233+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:34.237 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:34.237+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:34.416 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:34.416+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:35.217 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:35.216+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:35.266 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:35.264+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:35.444 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:35.444+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:36.264 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:36.264+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:36.307 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:36.307+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:36.449 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:36.449+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:37.238 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:37.238+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:37.265 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:37.264+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:37.483 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:37.483+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:38.234 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:38.234+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:38.237 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:38.236+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:38.438 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:38.439+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:39.242 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:39.243+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:39.257 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:39.258+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:39.431 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:39.432+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:40.216 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:40.217+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:40.250 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:40.250+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:40.475 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:40.475+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:41.203 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:41.203+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:41.204 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:41.205+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:41.465 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:41.466+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:42.164 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:42.164+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:42.166 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:42.167+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:42.493 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:42.493+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:43.126 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:43.126+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:43.136 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:43.137+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:43.531 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:43.531+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:44.079 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:44.080+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:44.088 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:44.089+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:44.517 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:44.517+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:45.062 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:45.063+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:45.094 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:45.094+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:45.545 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:45.546+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:46.067 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:46.068+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:46.116 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:46.117+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:46.542 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:46.543+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:47.025 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:47.025+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:47.102 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:47.103+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:47.527 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:47.528+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:47.976 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:47.977+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:48.142 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:48.143+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:48.533 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:48.533+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:48.950 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:48.951+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:49.167 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:49.168+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:49.486 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:49.487+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:49.984 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:49.984+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:50.142 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:50.143+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:50.461 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:50.461+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:50.978 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:50.979+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:51.133 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:51.134+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:51.496 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:51.496+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:51.960 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:51.959+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:52.165 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:52.165+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:52.462 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:52.462+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:52.934 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:52.932+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:53.154 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:53.153+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:53.432 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:53.432+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:53.914 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:53.914+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:54.120 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:54.120+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:54.449 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:54.449+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:54.869 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:54.869+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:55.124 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:55.124+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:55.498 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:55.497+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:55.870 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:55.870+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:56.090 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:56.089+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:56.515 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:56.514+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:56.881 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:56.881+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:57.098 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:57.097+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:57.523 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:57.523+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:57.880 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:57.880+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:58.070 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:58.070+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:58.572 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:58.571+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:58.890 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:58.890+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:59.098 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:49:59.097+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:59.549 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:49:59.549+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:49:59.907 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:49:59.907+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:00.136 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:00.135+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:00.506 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:00.505+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:00.864 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:00.864+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:01.178 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:01.178+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:01.549 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:01.548+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:01.910 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:01.909+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:02.154 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:02.153+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:02.529 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:02.528+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:02.922 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:02.922+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:03.153 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:03.152+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:03.487 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:03.487+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:03.969 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:03.969+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:04.125 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:04.125+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:04.527 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:04.527+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:04.956 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:04.955+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:05.082 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:05.082+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:05.541 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:05.541+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:05.996 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:05.996+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:06.060 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:06.060+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:06.553 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:06.552+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:07.007 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:07.006+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:07.014 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:07.014+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:07.544 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:07.544+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:07.978 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:07.978+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:08.030 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:08.029+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:08.512 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:08.511+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:08.936 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:08.936+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:08.992 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:08.992+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:09.464 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:09.464+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:09.916 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:09.916+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:09.994 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:09.993+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:10.506 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:10.505+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:10.876 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:10.876+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:10.961 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:10.960+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:11.555 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:11.555+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:11.856 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:11.856+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:11.985 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:11.984+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:12.572 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:12.572+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:12.882 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:12.882+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:12.984 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:12.984+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:13.562 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:13.561+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:13.851 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:13.851+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:14.014 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:14.014+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:14.577 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:14.576+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:14.896 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:14.896+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:15.055 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:15.055+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:15.533 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:15.533+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:15.894 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:15.894+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:16.058 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:16.057+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:16.580 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:16.580+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:16.846 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:16.845+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:17.086 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:17.086+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:17.567 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:17.567+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:17.896 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:17.895+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:18.119 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:18.118+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:18.612 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:18.611+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:18.890 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:18.889+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:19.132 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:19.132+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:19.583 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:19.584+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:19.872 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:19.872+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:20.085 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:20.085+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:20.571 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:20.572+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:20.900 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:20.900+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:21.086 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:21.085+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:21.544 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:21.544+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:21.859 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:21.859+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:22.118 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:22.119+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:22.586 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:22.587+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:22.898 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:22.899+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:23.107 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:23.107+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:23.550 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:23.550+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:23.882 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:23.883+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:24.087 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:24.087+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:24.555 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:24.556+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:24.912 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:24.913+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:25.074 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:25.075+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:25.530 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:25.530+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:25.893 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:25.893+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:26.042 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:26.043+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:26.542 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:26.542+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:26.892 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:26.893+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:27.041 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:27.041+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:27.592 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:27.592+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:27.943 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:27.943+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:28.091 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:28.091+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:28.641 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:28.641+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:28.944 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:28.945+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:29.087 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:29.088+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:29.646 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:29.647+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:29.927 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:29.928+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:30.046 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:30.047+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:30.642 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:30.642+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:30.911 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:30.912+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:31.038 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:31.038+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:31.651 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:31.652+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:31.912 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:31.913+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:32.086 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:32.087+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:32.639 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:32.640+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:32.896 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:32.896+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:33.086 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:33.087+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:33.640 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:33.641+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:33.908 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:33.909+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:34.090 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:34.091+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:34.591 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:34.592+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:34.912 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:34.912+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:35.074 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:35.074+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:35.622 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:35.622+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:35.893 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:35.893+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:36.078 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:36.078+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:36.577 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:36.577+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:36.884 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:36.884+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:37.096 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:37.096+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:37.559 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:37.559+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:37.858 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:37.858+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:38.104 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:38.104+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:38.531 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:38.531+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:38.874 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:38.874+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:39.150 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:39.150+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:39.535 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:39.534+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:39.832 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:39.832+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:40.182 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:40.181+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:40.502 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:40.502+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:40.843 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:40.843+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:41.142 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:41.142+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:41.513 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:41.513+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:41.864 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:41.864+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:42.110 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:42.110+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:42.504 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:42.504+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:42.819 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:42.819+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:43.136 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:43.136+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:43.498 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:43.498+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:43.857 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:43.857+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:44.174 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:44.174+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:44.521 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:44.521+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:44.839 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:44.839+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:45.198 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:45.197+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:45.505 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:45.505+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:45.835 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:45.835+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:46.233 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:46.232+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:46.483 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:46.483+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:46.840 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:46.840+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:47.266 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:47.266+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:47.530 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:47.530+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:47.824 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:47.824+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:48.303 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:48.303+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:48.534 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:48.533+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:48.782 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:48.782+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:49.290 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:49.289+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:49.555 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:49.555+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:49.797 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:49.797+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:50.248 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:50.248+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:50.562 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:50.562+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:50.839 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:50.839+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:51.240 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:51.240+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:51.609 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:51.608+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:51.860 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:51.860+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:52.278 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:52.278+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:52.623 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:52.623+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:52.839 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:52.838+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:53.266 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:53.265+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:53.660 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:53.660+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:53.826 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:53.826+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:54.298 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:54.298+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:54.691 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:54.691+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:54.798 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:54.798+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:55.326 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:55.325+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:55.681 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:55.681+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:55.759 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:55.759+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:56.291 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:56.291+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:56.649 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:56.649+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:56.717 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:56.717+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:57.277 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:57.276+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:57.621 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:57.620+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:57.748 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:57.748+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:58.280 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:58.280+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:58.669 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:58.668+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:58.789 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:58.789+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:59.294 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:50:59.294+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:59.670 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:50:59.670+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:50:59.802 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:50:59.802+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:00.255 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:00.255+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:00.702 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:00.702+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:00.808 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:00.808+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:01.215 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:01.215+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:01.744 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:01.744+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:01.796 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:01.796+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:02.206 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:02.206+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:02.754 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:02.754+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:02.847 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:02.846+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:03.245 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:03.245+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:03.779 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:03.780+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:03.810 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:03.810+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:04.286 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:04.286+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:04.744 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:04.744+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:04.831 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:04.832+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:05.333 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:05.333+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:05.727 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:05.728+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:05.871 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:05.872+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:06.380 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:06.380+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:06.705 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:06.705+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:06.847 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:06.848+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:07.400 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:07.401+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:07.720 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:07.721+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:07.841 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:07.841+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:08.413 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:08.413+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:08.684 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:08.684+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:08.841 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:08.841+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:09.452 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:09.453+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:09.642 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:09.643+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:09.851 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:09.852+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:10.425 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:10.426+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:10.602 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:10.603+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:10.879 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:10.880+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:11.405 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:11.405+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:11.634 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:11.635+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:11.861 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:11.862+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:12.387 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:12.388+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:12.681 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:12.682+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:12.819 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:12.819+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:13.396 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:13.396+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:13.656 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:13.657+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:13.855 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:13.856+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:14.399 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:14.399+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:14.672 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:14.672+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:14.829 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:14.830+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:15.402 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:15.402+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:15.690 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:15.690+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:15.872 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:15.872+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:16.424 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:16.425+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:16.693 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:16.693+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:16.866 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:16.867+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:17.442 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:17.443+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:17.712 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:17.713+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:17.906 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:17.907+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:18.398 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:18.399+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:18.714 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:18.714+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:18.917 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:18.917+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:19.375 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:19.375+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:19.764 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:19.763+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:19.960 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:19.960+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:20.380 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:20.380+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:20.752 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:20.751+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:20.971 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:20.971+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:21.351 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:21.351+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:21.710 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:21.710+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:21.959 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:21.959+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:22.366 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:22.366+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:22.706 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:22.706+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:22.927 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:22.927+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:23.329 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:23.329+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:23.667 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:23.667+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:23.954 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:23.954+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:24.379 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:24.379+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:24.693 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:24.692+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:25.000 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:24.999+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:25.355 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:25.355+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:25.686 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:25.685+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:25.987 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:25.987+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:26.337 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:26.336+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:26.696 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:26.695+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:26.994 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:26.994+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:27.292 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:27.291+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:27.737 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:27.737+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:28.021 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:28.021+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:28.340 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:28.340+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:28.692 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:28.692+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:28.981 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:28.980+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:29.346 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:29.345+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:29.672 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:29.671+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:29.940 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:29.940+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:30.323 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:30.323+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:30.686 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:30.686+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:30.953 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:30.952+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:31.372 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:31.371+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:31.692 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:31.691+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:31.908 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:31.908+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:32.408 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:32.407+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:32.736 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:32.736+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:32.956 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:32.956+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:33.369 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:33.369+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:33.688 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:33.687+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:33.983 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:33.983+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:34.352 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:34.351+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:34.674 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:34.674+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:35.007 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:35.007+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:35.395 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:35.395+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:35.685 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:35.685+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:36.032 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:36.032+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:36.413 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:36.412+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:36.692 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:36.692+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:37.026 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:37.026+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:37.414 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:37.414+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:37.650 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:37.650+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:38.029 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:38.028+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:38.391 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:38.391+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:38.640 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:38.639+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:39.042 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:39.042+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:39.392 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:39.392+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:39.606 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:39.606+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:40.069 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:40.069+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:40.400 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:40.400+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:40.582 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:40.582+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:41.044 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:41.044+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:41.383 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:41.383+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:41.563 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:41.563+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:42.069 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:42.069+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:42.382 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:42.381+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:42.545 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:42.544+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:43.064 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:43.064+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:43.413 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:43.413+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:43.538 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:43.538+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:44.104 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:44.103+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:44.437 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:44.436+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:44.525 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:44.525+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:45.103 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:45.103+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:45.457 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:45.456+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:45.478 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:45.477+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:46.090 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:46.090+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:46.465 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:46.465+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:46.487 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:46.487+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:47.093 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:47.093+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:47.417 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:47.416+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:47.529 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:47.530+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:48.084 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:48.084+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:48.386 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:48.386+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:48.552 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:48.552+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:49.071 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:49.071+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:49.412 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:49.412+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:49.599 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:49.600+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:50.086 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:50.086+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:50.450 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:50.451+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:50.561 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:50.561+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:51.081 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:51.082+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:51.486 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:51.487+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:51.540 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:51.541+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:52.035 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:52.036+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:52.520 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:52.521+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:52.569 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:52.569+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:53.017 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:53.017+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:53.500 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:53.501+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:53.587 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:53.588+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:54.028 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:54.029+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:54.524 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:54.525+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:54.635 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:54.635+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:55.034 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:55.034+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:55.512 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:55.513+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:55.649 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:55.650+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:56.070 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:56.071+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:56.525 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:56.526+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:56.638 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:56.639+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:57.049 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:57.050+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:57.494 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:57.495+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:57.644 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:57.644+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:58.080 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:58.081+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:58.507 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:58.507+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:58.678 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:58.679+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:59.105 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:51:59.106+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:59.548 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:51:59.549+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:51:59.728 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:51:59.729+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:00.103 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:00.103+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:00.540 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:00.541+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:00.720 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:00.721+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:01.056 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:01.057+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:01.585 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:01.585+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:01.711 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:01.712+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:02.060 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:02.061+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:02.612 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:02.610+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:02.662 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:02.661+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:03.080 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:03.080+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:03.580 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:03.580+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:03.651 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:03.650+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:04.049 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:04.048+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:04.556 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:04.556+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:04.603 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:04.603+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:05.093 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:05.093+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:05.538 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:05.538+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:05.637 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:05.636+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:06.102 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:06.102+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:06.505 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:06.505+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:06.590 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:06.589+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:07.089 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:07.088+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:07.472 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:07.471+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:07.580 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:07.580+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:08.097 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:08.097+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:08.522 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:08.522+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:08.553 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:08.552+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:09.113 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:09.112+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:09.506 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:09.506+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:09.552 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:09.552+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:10.086 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:10.086+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:10.549 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:10.549+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:10.574 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:10.574+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:11.131 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:11.131+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:11.529 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:11.529+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:11.620 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:11.619+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:12.159 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:12.159+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:12.538 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:12.537+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:12.631 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:12.630+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:13.145 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:13.144+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:13.563 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:13.563+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:13.600 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:13.599+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:14.166 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:14.165+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:14.530 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:14.530+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:14.621 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:14.620+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:15.190 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:15.189+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:15.511 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:15.511+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:15.657 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:15.656+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:16.240 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:16.240+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:16.547 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:16.547+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:16.641 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:16.640+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:17.242 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:17.242+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:17.569 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:17.569+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:17.645 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:17.645+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:18.230 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:18.230+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:18.524 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:18.524+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:18.614 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:18.613+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:19.185 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:19.185+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:19.527 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:19.526+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:19.603 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:19.602+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:20.169 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:20.169+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:20.503 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:20.502+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:20.583 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:20.582+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:21.185 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:21.185+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:21.533 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:21.533+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:21.573 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:21.572+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:22.156 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:22.156+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:22.497 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:22.495+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:22.614 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:22.614+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:23.167 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:23.167+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:23.450 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:23.450+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:23.579 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:23.578+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:24.158 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:24.158+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:24.478 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:24.478+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:24.552 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:24.551+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:25.196 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:25.196+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:25.431 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:25.431+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:25.572 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:25.572+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:26.178 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:26.178+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:26.482 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:26.481+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:26.590 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:26.590+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:27.175 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:27.175+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:27.510 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:27.510+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:27.633 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:27.633+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:28.164 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:28.164+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:28.516 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:28.515+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:28.651 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:28.651+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:29.195 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:29.194+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:29.553 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:29.553+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:29.648 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:29.648+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:30.148 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:30.148+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:30.576 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:30.576+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:30.626 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:30.626+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:31.197 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:31.196+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:31.542 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:31.541+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:31.580 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:31.580+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:32.159 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:32.158+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:32.555 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:32.555+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:32.574 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:32.574+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:33.112 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:33.113+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:33.526 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:33.527+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:33.577 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:33.577+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:34.070 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:34.071+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:34.526 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:34.526+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:34.538 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:34.539+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:35.032 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:35.033+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:35.554 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:35.554+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:35.589 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:35.590+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:36.006 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:36.006+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:36.517 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:36.518+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:36.555 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:36.556+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:37.013 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:37.013+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:37.540 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:37.541+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:37.563 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:37.564+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:38.056 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:38.057+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:38.522 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:38.522+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:38.551 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:38.552+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:39.060 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:39.061+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:39.514 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:39.515+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:39.554 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:39.555+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:40.020 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:40.021+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:40.495 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:40.495+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:40.580 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:40.581+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:41.061 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:41.061+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:41.524 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:41.525+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:41.554 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:41.555+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:42.099 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:42.100+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:42.488 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:42.489+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:42.582 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:42.583+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:43.136 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:43.137+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:43.501 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:43.502+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:43.632 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:43.632+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:44.088 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:44.089+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:44.452 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:44.452+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:44.679 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:44.680+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:45.133 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:45.134+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:45.498 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:45.499+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:45.652 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:45.652+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:46.104 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:46.103+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:46.458 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:46.458+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:46.658 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:46.658+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:47.119 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:47.118+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:47.458 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:47.458+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:47.657 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:47.657+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:48.126 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:48.125+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:48.437 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:48.436+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:48.611 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:48.610+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:49.123 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:49.122+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:49.422 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:49.422+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:49.581 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:49.581+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:50.142 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:50.142+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:50.413 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:50.413+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:50.553 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:50.552+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:51.149 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:51.148+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:51.444 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:51.443+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:51.563 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:51.562+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:52.141 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:52.140+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:52.455 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:52.454+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:52.520 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:52.519+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:53.111 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:53.110+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:53.465 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:53.465+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:53.478 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:53.478+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:54.097 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:54.096+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:54.431 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:54.431+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:54.467 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:54.467+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:55.127 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:55.127+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:55.462 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:55.462+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:55.471 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:55.471+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:56.145 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:56.145+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:56.432 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:56.432+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:56.480 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:56.480+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:57.189 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:57.188+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:57.479 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:57.479+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:57.498 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:57.497+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:58.180 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:58.180+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:58.504 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:58.504+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:58.539 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:58.539+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:59.205 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:52:59.204+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:59.490 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:52:59.490+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:52:59.523 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:52:59.523+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:00.192 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:00.192+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:00.504 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:00.504+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:00.533 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:00.533+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:01.168 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:01.168+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:01.466 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:01.466+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:01.484 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:01.484+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:02.175 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:02.175+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:02.441 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:02.440+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:02.446 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:02.446+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:03.140 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:03.140+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:03.416 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:03.416+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:03.420 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:03.420+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:04.157 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:04.157+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:04.405 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:04.404+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:04.464 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:04.464+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:05.196 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:05.196+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:05.433 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:05.432+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:05.496 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:05.496+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:06.237 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:06.236+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:06.445 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:06.445+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:06.486 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:06.486+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:07.230 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:07.230+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:07.451 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:07.450+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:07.468 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:07.468+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:08.279 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:08.278+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:08.429 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:08.428+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:08.442 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:08.442+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:09.238 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:09.238+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:09.426 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:09.426+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:09.437 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:09.436+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:10.212 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:10.212+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:10.384 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:10.384+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:10.418 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:10.417+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:11.190 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:11.189+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:11.350 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:11.350+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:11.388 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:11.388+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:12.221 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:12.221+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:12.352 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:12.352+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:12.362 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:12.361+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:13.180 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:13.180+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:13.378 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:13.378+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:13.405 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:13.405+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:14.180 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:14.180+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:14.355 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:14.356+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:14.378 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:14.377+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:15.218 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:15.218+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:15.331 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:15.331+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:15.356 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:15.357+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:16.226 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:16.226+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:16.334 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:16.334+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:16.360 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:16.360+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:17.200 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:17.201+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:17.313 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:17.314+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:17.370 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:17.370+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:18.164 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:18.165+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:18.322 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:18.323+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:18.380 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:18.381+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:19.124 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:19.124+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:19.361 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:19.362+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:19.407 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:19.408+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:20.115 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:20.115+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:20.345 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:20.346+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:20.446 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:20.447+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:21.134 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:21.134+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:21.335 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:21.336+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:21.451 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:21.452+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:22.175 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:22.174+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:22.322 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:22.323+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:22.436 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:22.436+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:23.170 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:23.170+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:23.277 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:23.278+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:23.400 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:23.400+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:24.158 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:24.159+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:24.280 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:24.281+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:24.385 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:24.385+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:25.188 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:25.189+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:25.293 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:25.294+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:25.335 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:25.336+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:26.141 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:26.142+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:26.251 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:26.251+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:26.360 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:26.361+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:27.170 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:27.170+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:27.218 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:27.203+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:27.359 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:27.360+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:28.156 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:28.156+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:28.219 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:28.220+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:28.381 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:28.382+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:29.113 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:29.114+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:29.262 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:29.262+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:29.425 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:29.425+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:30.163 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:30.163+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:30.215 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:30.215+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:30.398 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:30.398+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:31.135 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:31.134+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:31.255 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:31.255+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:31.412 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:31.411+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:32.116 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:32.116+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:32.279 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:32.279+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:32.413 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:32.413+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:33.078 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:33.078+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:33.251 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:33.251+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:33.393 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:33.392+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:34.099 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:34.099+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:34.290 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:34.289+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:34.404 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:34.404+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:35.085 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:35.085+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:35.245 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:35.245+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:35.370 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:35.370+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:36.097 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:36.097+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:36.230 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:36.230+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:36.378 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:36.378+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:37.084 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:37.084+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:37.235 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:37.235+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:37.368 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:37.368+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:38.069 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:38.069+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:38.272 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:38.272+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:38.383 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:38.383+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:39.053 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:39.053+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:39.268 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:39.268+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:39.396 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:39.396+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:40.073 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:40.073+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:40.226 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:40.226+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:40.356 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:40.356+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:41.121 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:41.121+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:41.231 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:41.231+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:41.370 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:41.370+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:42.156 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:42.156+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:42.193 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:42.193+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:42.402 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:42.401+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:43.156 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:43.156+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:43.201 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:43.200+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:43.406 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:43.405+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:44.133 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:44.133+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:44.179 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:44.179+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:44.378 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:44.377+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:45.158 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:45.158+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:45.173 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:45.173+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:45.408 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:45.408+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:46.129 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:46.129+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:46.156 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:46.155+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:46.413 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:46.412+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:47.104 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:47.104+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:47.154 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:47.154+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:47.389 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:47.389+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:48.105 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:48.105+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:48.119 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:48.119+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:48.374 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:48.374+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:49.103 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:49.102+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:49.109 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:49.109+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:49.346 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:49.346+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:50.099 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:50.098+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:50.107 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:50.106+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:50.394 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:50.393+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:51.055 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:51.055+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:51.093 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:51.093+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:51.348 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:51.348+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:52.047 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:52.047+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:52.079 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:52.078+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:52.355 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:52.355+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:53.072 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:53.072+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:53.077 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:53.077+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:53.360 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:53.360+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:54.107 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:54.106+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:54.107 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:54.107+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:54.336 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:54.336+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:55.061 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:55.061+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:55.095 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:55.095+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:55.328 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:55.328+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:56.065 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:56.064+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:56.090 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:56.089+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:56.357 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:56.357+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:57.071 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:57.071+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:57.112 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:57.112+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:57.383 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:57.383+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:58.053 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:58.053+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:58.155 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:58.155+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:58.336 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:58.336+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:59.047 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:53:59.047+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:59.168 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:53:59.168+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:53:59.311 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:53:59.311+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:00.058 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:00.057+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:00.175 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:00.175+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:00.336 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:00.336+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:01.070 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:01.071+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:01.204 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:01.205+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:01.362 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:01.363+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:02.078 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:02.079+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:02.247 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:02.248+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:02.385 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:02.386+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:03.039 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:03.040+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:03.274 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:03.275+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:03.365 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:03.366+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:04.077 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:04.078+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:04.271 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:04.272+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:04.328 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:04.329+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:05.112 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:05.113+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:05.318 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:05.319+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:05.350 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:05.351+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:06.099 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:06.100+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:06.349 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:06.350+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:06.367 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:06.368+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:07.060 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:07.061+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:07.339 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:07.340+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:07.377 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:07.377+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:08.086 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:08.087+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:08.356 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:08.357+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:08.372 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:08.373+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:09.045 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:09.046+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:09.335 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:09.336+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:09.399 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:09.399+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:10.037 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:10.037+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:10.326 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:10.327+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:10.401 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:10.401+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:11.080 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:11.081+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:11.291 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:11.292+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:11.351 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:11.352+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:12.046 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:12.047+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:12.315 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:12.316+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:12.355 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:12.356+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:13.023 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:13.023+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:13.276 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:13.276+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:13.317 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:13.318+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:14.058 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:14.058+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:14.259 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:14.259+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:14.288 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:14.287+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:15.107 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:15.106+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:15.216 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:15.216+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:15.297 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:15.294+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:16.089 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:16.088+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:16.238 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:16.238+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:16.258 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:16.257+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:17.126 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:17.126+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:17.199 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:17.199+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:17.259 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:17.259+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:18.103 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:18.103+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:18.216 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:18.215+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:18.216 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:18.216+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:19.068 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:19.068+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:19.179 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:19.179+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:19.252 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:19.252+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:20.070 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:20.070+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:20.160 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:20.160+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:20.262 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:20.262+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:21.106 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:21.106+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:21.151 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:21.151+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:21.268 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:21.267+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:22.101 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:22.101+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:22.117 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:22.117+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:22.304 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:22.304+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:23.084 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:23.084+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:23.131 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:23.131+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:23.281 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:23.280+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:24.035 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:24.034+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:24.122 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:24.122+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:24.259 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:24.258+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:25.057 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:25.057+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:25.129 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:25.129+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:25.297 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:25.296+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:26.081 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:26.081+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:26.118 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:26.117+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:26.325 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:26.324+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:27.088 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:27.087+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:27.137 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:27.137+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:27.286 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:27.286+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:28.063 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:28.063+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:28.162 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:28.162+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:28.277 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:28.277+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:29.087 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:29.087+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:29.196 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:29.196+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:29.286 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:29.286+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:30.131 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:30.131+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:30.232 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:30.231+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:30.312 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:30.312+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:31.139 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:31.138+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:31.231 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:31.231+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:31.286 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:31.286+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:32.128 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:32.128+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:32.238 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:32.237+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:32.278 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:32.278+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:33.152 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:33.152+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:33.267 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:33.267+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:33.285 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:33.285+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:34.135 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:34.134+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:34.252 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:34.252+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:34.256 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:34.256+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:35.123 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:35.123+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:35.225 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:35.225+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:35.242 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:35.242+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:36.126 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:36.126+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:36.186 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:36.186+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:36.235 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:36.235+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:37.158 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:37.158+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:37.190 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:37.189+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:37.221 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:37.221+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:38.199 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:38.199+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:38.205 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:38.205+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:38.208 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:38.208+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:39.176 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:39.176+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:39.182 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:39.182+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:39.190 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:39.190+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:40.183 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:40.183+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:40.195 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:40.194+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:40.210 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:40.210+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:41.165 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:41.164+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:41.189 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:41.189+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:41.225 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:41.225+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:42.122 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:42.122+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:42.215 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:42.214+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:42.215 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:42.214+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:43.120 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:43.120+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:43.167 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:43.167+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:43.240 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:43.240+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:44.147 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:44.147+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:44.157 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:44.157+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:44.233 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:44.234+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:45.135 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:45.135+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:45.201 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:45.202+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:45.282 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:45.283+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:46.093 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:46.094+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:46.210 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:46.211+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:46.283 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:46.283+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:47.087 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:47.088+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:47.167 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:47.168+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:47.289 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:47.289+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:48.107 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:48.108+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:48.215 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:48.216+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:48.337 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:48.338+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:49.158 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:49.158+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:49.202 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:49.201+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:49.318 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:49.319+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:50.155 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:50.155+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:50.227 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:50.227+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:50.342 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:50.343+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:51.144 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:51.145+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:51.190 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:51.190+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:51.325 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:51.326+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:52.107 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:52.107+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:52.187 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:52.188+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:52.301 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:52.302+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:53.097 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:53.098+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:53.173 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:53.173+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:53.311 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:53.312+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:54.130 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:54.130+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:54.165 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:54.166+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:54.286 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:54.286+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:55.084 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:55.085+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:55.161 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:55.162+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:55.306 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:55.307+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:56.096 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:56.097+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:56.145 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:56.146+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:56.349 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:56.350+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:57.103 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:57.104+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:57.138 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:57.139+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:57.351 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:57.352+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:58.107 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:58.107+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:58.132 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:58.131+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:58.349 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:58.349+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:59.089 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:54:59.089+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:59.131 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:54:59.131+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:54:59.352 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:54:59.352+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:00.083 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:00.083+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:00.139 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:00.139+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:00.309 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:00.308+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:01.126 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:01.126+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:01.136 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:01.135+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:01.345 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:01.345+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:02.124 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:02.123+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:02.142 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:02.142+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:02.307 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:02.307+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:03.089 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:03.089+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:03.152 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:03.152+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:03.322 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:03.321+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:04.127 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:04.127+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:04.139 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:04.139+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:04.357 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:04.356+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:05.118 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:05.117+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:05.118 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:05.118+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:05.331 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:05.331+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:06.133 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:06.133+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:06.142 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:06.142+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:06.375 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:06.375+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:07.093 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:07.093+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:07.174 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:07.174+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:07.379 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:07.379+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:08.051 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:08.051+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:08.124 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:08.124+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:08.356 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:08.356+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:09.072 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:09.072+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:09.080 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:09.080+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:09.353 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:09.353+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:10.057 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:10.057+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:10.108 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:10.107+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:10.334 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:10.334+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:11.058 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:11.058+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:11.087 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:11.087+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:11.347 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:11.347+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:12.029 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:12.028+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:12.130 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:12.130+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:12.388 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:12.388+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:13.078 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:13.078+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:13.149 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:13.149+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:13.362 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:13.361+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:14.066 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:14.066+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:14.129 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:14.129+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:14.379 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:14.378+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:15.087 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:15.087+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:15.171 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:15.170+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:15.393 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:15.393+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:16.039 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:16.039+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:16.152 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:16.152+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:16.409 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:16.409+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:17.042 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:17.042+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:17.110 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:17.110+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:17.411 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:17.411+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:18.070 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:18.069+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:18.125 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:18.125+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:18.405 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:18.405+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:19.111 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:19.111+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:19.119 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:19.119+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:19.381 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:19.381+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:20.143 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:20.142+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:20.145 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:20.145+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:20.400 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:20.400+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:21.133 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:21.133+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:21.141 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:21.141+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:21.421 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:21.421+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:22.108 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:22.107+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:22.159 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:22.159+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:22.439 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:22.439+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:23.067 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:23.066+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:23.207 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:23.207+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:23.466 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:23.465+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:24.026 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:24.026+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:24.186 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:24.185+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:24.483 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:24.483+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:24.984 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:24.984+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:25.189 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:25.189+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:25.476 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:25.476+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:25.949 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:25.949+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:26.167 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:26.167+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:26.438 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:26.438+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:26.933 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:26.932+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:27.200 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:27.200+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:27.437 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:27.437+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:27.922 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:27.922+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:28.175 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:28.175+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:28.464 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:28.464+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:28.885 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:28.885+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:29.224 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:29.224+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:29.439 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:29.440+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:29.856 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:29.856+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:30.233 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:30.234+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:30.441 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:30.442+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:30.896 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:30.896+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:31.281 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:31.281+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:31.410 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:31.411+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:31.883 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:31.883+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:32.323 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:32.324+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:32.451 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:32.452+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:32.901 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:32.902+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:33.326 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:33.327+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:33.406 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:33.407+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:33.871 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:33.872+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:34.304 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:34.305+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:34.359 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:34.360+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:34.828 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:34.828+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:35.341 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:35.342+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:35.352 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:35.352+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:35.778 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:35.779+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:36.293 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:36.294+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:36.369 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:36.370+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:36.776 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:36.776+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:37.281 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:37.282+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:37.349 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:37.350+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:37.798 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:37.799+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:38.331 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:38.332+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:38.339 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:38.340+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:38.811 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:38.811+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:39.296 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:39.297+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:39.328 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:39.329+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:39.792 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:39.793+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:40.267 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:40.268+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:40.298 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:40.298+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:40.806 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:40.807+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:41.255 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:41.256+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:41.271 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:41.272+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:41.848 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:41.848+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:42.279 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:42.279+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:42.311 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:42.311+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:42.856 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:42.856+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:43.242 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:43.242+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:43.337 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:43.336+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:43.885 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:43.883+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:44.269 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:44.268+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:44.382 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:44.381+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:44.846 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:44.846+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:45.233 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:45.233+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:45.387 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:45.387+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:45.838 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:45.838+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:46.225 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:46.225+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:46.347 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:46.347+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:46.826 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:46.826+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:47.231 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:47.231+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:47.339 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:47.338+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:47.803 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:47.802+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:48.189 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:48.189+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:48.367 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:48.366+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:48.840 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:48.840+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:49.155 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:49.155+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:49.327 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:49.327+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:49.857 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:49.857+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:50.171 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:50.171+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:50.375 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:50.374+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:50.814 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:50.814+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:51.150 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:51.150+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:51.341 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:51.341+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:51.805 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:51.804+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:52.154 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:52.153+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:52.363 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:52.363+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:52.836 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:52.836+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:53.141 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:53.140+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:53.368 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:53.368+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:53.792 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:53.792+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:54.098 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:54.098+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:54.372 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:54.371+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:54.832 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:54.832+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:55.118 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:55.118+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:55.354 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:55.353+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:55.879 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:55.878+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:56.147 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:56.146+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:56.352 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:56.351+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:56.896 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:56.895+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:57.118 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:57.117+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:57.385 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:57.385+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:57.907 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:57.906+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:58.107 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:58.107+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:58.367 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:58.367+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:58.898 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:58.898+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:59.150 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:55:59.149+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:59.415 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:55:59.415+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:55:59.911 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:55:59.911+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:00.165 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:00.165+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:00.448 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:00.447+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:00.896 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:00.896+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:01.140 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:01.140+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:01.480 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:01.480+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:01.922 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:01.922+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:02.135 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:02.134+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:02.437 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:02.437+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:02.911 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:02.910+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:03.179 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:03.178+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:03.461 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:03.461+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:03.882 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:03.882+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:04.186 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:04.185+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:04.443 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:04.443+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:04.916 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:04.916+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:05.140 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:05.140+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:05.408 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:05.408+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:05.872 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:05.872+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:06.095 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:06.095+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:06.369 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:06.369+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:06.890 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:06.890+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:07.139 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:07.139+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:07.382 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:07.382+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:07.858 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:07.858+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:08.148 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:08.147+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:08.432 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:08.432+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:08.899 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:08.899+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:09.104 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:09.104+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:09.417 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:09.417+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:09.925 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:09.925+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:10.118 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:10.117+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:10.378 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:10.378+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:10.942 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:10.941+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:11.149 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:11.149+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:11.404 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:11.404+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:11.979 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:11.980+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:12.158 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:12.157+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:12.387 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:12.387+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:12.942 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:12.942+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:13.144 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:13.143+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:13.430 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:13.431+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:13.979 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:13.979+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:14.120 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:14.121+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:14.403 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:14.404+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:14.956 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:14.957+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:15.154 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:15.155+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:15.364 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:15.364+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:15.946 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:15.946+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:16.166 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:16.167+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:16.325 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:16.326+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:16.942 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:16.943+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:17.178 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:17.179+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:17.300 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:17.301+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:17.981 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:17.982+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:18.154 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:18.155+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:18.252 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:18.253+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:19.022 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:19.023+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:19.166 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:19.167+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:19.252 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:19.253+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:20.043 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:20.044+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:20.126 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:20.127+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:20.231 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:20.232+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:21.037 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:21.038+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:21.128 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:21.129+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:21.185 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:21.186+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:22.007 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:22.008+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:22.165 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:22.166+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:22.198 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:22.199+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:23.055 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:23.056+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:23.174 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:23.175+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:23.187 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:23.188+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:24.069 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:24.070+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:24.136 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:24.137+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:24.237 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:24.238+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:25.108 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:25.109+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:25.154 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:25.154+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:25.277 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:25.278+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:26.068 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:26.069+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:26.106 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:26.107+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:26.307 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:26.308+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:27.084 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:27.084+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:27.113 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:27.112+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:27.308 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:27.307+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:28.052 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:28.052+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:28.121 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:28.121+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:28.330 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:28.330+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:29.102 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:29.102+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:29.103 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:29.103+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:29.338 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:29.338+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:30.095 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:30.095+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:30.122 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:30.122+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:30.291 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:30.291+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:31.124 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:31.123+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:31.134 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:31.134+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:31.288 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:31.288+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:32.116 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:32.115+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:32.139 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:32.139+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:32.308 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:32.308+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:33.111 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:33.110+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:33.158 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:33.157+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:33.314 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:33.313+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:34.066 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:34.066+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:34.178 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:34.178+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:34.316 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:34.315+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:35.053 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:35.052+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:35.166 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:35.166+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:35.318 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:35.317+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:36.017 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:36.016+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:36.189 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:36.189+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:36.279 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:36.279+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:37.027 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:37.026+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:37.232 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:37.232+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:37.302 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:37.302+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:38.052 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:38.052+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:38.277 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:38.276+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:38.284 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:38.284+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:39.026 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:39.026+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:39.256 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:39.256+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:39.323 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:39.323+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:40.064 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:40.064+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:40.284 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:40.284+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:40.300 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:40.300+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:41.078 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:41.078+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:41.264 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:41.263+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:41.302 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:41.302+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:42.117 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:42.117+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:42.226 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:42.226+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:42.305 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:42.305+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:43.085 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:43.085+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:43.256 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:43.256+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:43.343 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:43.342+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:44.081 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:44.081+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:44.248 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:44.248+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:44.379 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:44.379+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:45.047 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:45.047+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:45.282 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:45.281+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:45.410 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:45.409+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:46.055 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:46.054+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:46.265 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:46.265+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:46.447 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:46.447+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:47.100 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:47.100+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:47.220 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:47.220+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:47.451 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:47.450+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:48.068 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:48.068+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:48.230 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:48.229+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:48.458 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:48.458+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:49.116 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:49.116+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:49.191 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:49.190+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:49.451 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:49.451+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:50.159 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:50.159+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:50.205 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:50.205+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:50.408 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:50.408+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:51.157 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:51.157+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:51.244 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:51.244+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:51.456 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:51.456+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:52.149 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:52.148+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:52.293 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:52.292+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:52.446 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:52.446+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:53.144 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:53.144+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:53.330 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:53.330+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:53.468 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:53.468+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:54.128 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:54.128+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:54.312 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:54.311+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:54.434 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:54.434+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:55.092 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:55.092+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:55.329 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:55.329+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:55.387 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:55.387+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:56.084 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:56.084+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:56.292 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:56.292+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:56.422 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:56.422+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:57.083 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:57.083+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:57.246 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:57.246+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:57.379 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:57.379+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:58.110 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:58.111+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:58.200 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:58.200+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:58.346 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:58.336+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:59.155 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:56:59.156+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:59.213 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:56:59.214+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:56:59.373 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:56:59.374+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:00.116 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:00.116+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:00.253 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:00.254+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:00.393 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:00.393+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:01.082 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:01.083+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:01.289 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:01.290+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:01.419 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:01.420+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:02.083 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:02.084+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:02.295 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:02.296+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:02.376 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:02.377+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:03.043 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:03.044+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:03.251 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:03.251+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:03.410 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:03.411+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:04.056 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:04.057+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:04.261 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:04.262+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:04.437 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:04.438+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:05.064 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:05.065+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:05.266 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:05.267+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:05.426 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:05.427+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:06.077 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:06.077+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:06.315 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:06.316+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:06.407 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:06.407+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:07.088 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:07.089+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:07.365 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:07.365+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:07.454 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:07.455+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:08.115 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:08.115+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:08.332 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:08.333+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:08.499 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:08.500+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:09.082 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:09.083+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:09.373 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:09.374+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:09.519 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:09.520+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:10.053 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:10.054+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:10.349 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:10.349+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:10.567 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:10.568+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:11.067 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:11.067+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:11.321 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:11.321+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:11.583 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:11.583+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:12.041 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:12.041+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:12.357 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:12.356+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:12.550 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:12.550+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:13.091 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:13.090+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:13.361 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:13.360+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:13.504 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:13.502+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:14.097 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:14.096+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:14.378 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:14.377+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:14.499 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:14.499+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:15.127 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:15.127+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:15.401 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:15.401+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:15.523 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:15.523+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:16.129 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:16.129+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:16.437 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:16.437+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:16.532 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:16.532+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:17.112 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:17.111+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:17.405 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:17.404+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:17.552 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:17.551+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:18.108 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:18.108+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:18.373 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:18.373+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:18.582 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:18.581+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:19.102 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:19.101+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:19.364 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:19.364+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:19.605 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:19.605+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:20.097 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:20.097+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:20.375 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:20.374+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:20.650 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:20.650+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:21.050 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:21.050+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:21.398 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:21.397+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:21.700 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:21.699+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:22.056 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:22.056+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:22.411 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:22.411+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:22.727 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:22.727+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:23.014 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:23.014+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:23.444 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:23.444+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:23.719 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:23.719+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:24.052 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:24.052+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:24.397 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:24.397+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:24.733 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:24.732+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:25.035 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:25.035+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:25.417 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:25.417+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:25.768 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:25.767+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:26.012 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:26.011+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:26.461 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:26.461+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:26.770 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:26.770+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:27.004 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:27.004+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:27.474 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:27.473+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:27.803 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:27.802+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:28.035 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:28.034+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:28.514 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:28.513+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:28.826 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:28.826+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:29.030 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:29.030+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:29.474 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:29.473+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:29.841 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:29.841+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:30.007 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:30.007+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:30.500 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:30.499+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:30.879 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:30.879+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:31.040 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:31.039+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:31.500 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:31.500+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:31.897 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:31.897+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:32.003 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:32.003+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:32.528 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:32.528+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:32.947 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:32.947+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:32.953 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:32.953+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:33.485 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:33.485+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:33.946 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:33.946+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:33.970 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:33.970+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:34.515 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:34.514+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:34.969 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:34.968+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:35.021 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:35.021+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:35.537 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:35.537+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:35.937 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:35.936+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:36.023 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:36.023+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:36.555 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:36.555+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:36.888 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:36.887+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:36.978 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:36.978+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:37.545 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:37.545+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:37.932 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:37.932+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:37.953 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:37.953+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:38.529 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:38.528+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:38.920 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:38.920+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:38.935 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:38.935+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:39.521 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:39.521+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:39.962 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:39.963+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:39.971 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:39.971+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:40.530 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:40.529+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:40.919 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:40.918+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:40.931 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:40.931+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:41.542 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:41.542+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:41.886 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:41.887+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:41.908 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:41.908+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:42.523 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:42.523+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:42.865 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:42.866+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:42.903 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:42.904+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:43.516 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:43.517+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:43.865 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:43.866+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:43.894 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:43.895+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:44.513 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:44.513+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:44.862 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:44.862+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:44.924 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:44.925+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:45.507 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:45.508+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:45.861 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:45.862+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:45.963 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:45.964+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:46.480 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:46.481+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:46.879 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:46.880+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:46.970 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:46.971+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:47.455 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:47.456+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:47.863 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:47.864+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:47.981 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:47.982+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:48.407 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:48.408+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:48.894 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:48.895+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:48.936 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:48.937+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:49.457 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:49.457+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:49.931 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:49.932+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:49.939 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:49.939+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:50.443 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:50.442+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:50.909 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:50.907+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:50.925 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:50.926+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:51.431 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:51.432+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:51.950 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:51.951+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:51.954 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:51.955+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:52.409 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:52.410+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:52.907 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:52.908+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:52.951 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:52.951+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:53.419 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:53.420+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:53.915 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:53.915+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:53.929 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:53.930+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:54.406 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:54.406+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:54.914 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:54.914+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:54.929 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:54.930+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:55.402 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:55.402+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:55.869 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:55.870+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:55.907 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:55.907+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:56.355 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:56.354+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:56.872 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:56.871+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:56.935 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:56.935+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:57.325 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:57.325+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:57.907 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:57.906+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:57.920 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:57.920+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:58.291 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:58.291+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:58.881 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:58.881+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:58.919 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:58.919+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:59.320 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:57:59.320+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:59.910 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:57:59.910+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:57:59.921 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:57:59.921+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:00.289 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:00.288+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:00.886 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:00.886+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:00.953 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:00.953+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:01.337 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:01.337+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:01.879 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:01.879+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:01.927 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:01.927+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:02.333 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:02.333+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:02.911 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:02.911+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:02.920 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:02.920+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:03.298 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:03.298+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:03.863 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:03.863+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:03.957 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:03.957+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:04.339 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:04.338+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:04.847 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:04.847+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:04.991 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:04.991+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:05.353 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:05.352+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:05.853 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:05.852+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:06.002 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:06.002+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:06.317 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:06.317+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:06.819 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:06.819+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:06.958 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:06.958+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:07.323 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:07.323+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:07.822 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:07.822+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:08.008 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:08.008+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:08.321 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:08.321+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:08.835 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:08.835+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:08.978 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:08.977+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:09.342 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:09.342+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:09.866 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:09.865+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:09.935 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:09.934+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:10.307 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:10.307+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:10.843 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:10.843+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:10.912 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:10.912+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:11.262 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:11.262+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:11.859 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:11.859+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:11.894 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:11.894+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:12.236 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:12.236+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:12.871 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:12.871+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:12.940 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:12.940+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:13.258 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:13.257+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:13.863 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:13.863+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:13.942 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:13.942+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:14.285 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:14.285+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:14.907 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:14.907+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:14.939 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:14.939+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:15.253 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:15.252+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:15.924 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:15.924+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:15.925 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:15.924+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:16.268 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:16.268+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:16.899 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:16.899+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:16.945 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:16.944+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:17.243 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:17.243+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:17.909 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:17.908+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:17.993 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:17.993+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:18.196 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:18.196+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:18.911 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:18.911+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:18.957 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:18.957+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:19.161 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:19.161+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:19.955 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:19.955+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:19.991 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:19.991+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:20.156 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:20.155+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:20.928 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:20.928+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:20.953 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:20.953+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:21.111 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:21.111+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:21.885 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:21.885+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:21.924 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:21.924+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:22.075 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:22.075+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:22.902 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:22.902+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:22.933 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:22.933+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:23.045 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:23.045+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:23.924 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:23.924+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:23.939 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:23.939+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:24.033 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:24.031+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:24.926 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:24.926+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:24.951 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:24.951+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:25.001 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:25.001+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:25.928 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:25.928+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:25.968 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:25.968+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:26.039 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:26.038+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:26.905 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:26.906+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:27.011 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:27.012+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:27.032 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:27.032+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:27.928 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:27.929+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:28.007 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:28.007+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:28.021 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:28.022+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:28.973 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:28.973+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:29.020 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:29.021+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:29.062 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:29.063+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:30.012 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:30.013+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:30.066 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:30.067+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:30.087 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:30.088+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:31.026 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:31.027+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:31.073 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:31.074+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:31.114 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:31.115+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:32.027 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:32.028+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:32.090 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:32.091+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:32.124 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:32.125+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:32.998 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:32.999+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:33.085 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:33.085+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:33.118 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:33.119+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:33.957 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:33.957+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:34.073 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:34.074+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:34.156 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:34.157+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:34.921 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:34.922+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:35.031 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:35.031+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:35.163 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:35.163+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:35.915 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:35.916+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:36.067 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:36.068+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:36.133 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:36.134+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:36.920 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:36.921+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:37.088 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:37.089+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:37.097 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:37.096+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:37.872 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:37.873+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:38.094 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:38.094+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:38.115 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:38.116+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:38.894 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:38.895+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:39.061 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:39.061+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:39.091 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:39.092+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:39.872 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:39.872+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:40.053 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:40.054+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:40.110 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:40.111+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:40.897 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:40.897+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:41.089 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:41.088+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:41.152 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:41.152+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:41.926 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:41.926+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:42.046 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:42.046+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:42.167 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:42.166+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:42.902 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:42.902+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:43.001 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:43.000+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:43.128 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:43.128+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:43.945 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:43.945+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:43.965 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:43.964+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:44.116 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:44.116+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:44.943 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:44.942+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:44.963 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:44.962+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:45.114 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:45.114+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:45.896 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:45.896+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:45.958 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:45.957+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:46.128 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:46.128+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:46.888 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:46.888+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:46.946 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:46.945+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:47.108 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:47.108+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:47.906 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:47.906+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:47.926 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:47.926+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:48.083 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:48.082+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:48.877 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:48.877+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:48.970 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:48.970+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:49.086 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:49.086+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:49.920 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:49.920+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:49.978 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:49.978+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:50.064 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:50.064+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:50.942 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:50.942+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:50.978 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:50.978+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:51.078 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:51.078+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:51.969 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:51.969+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:52.004 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:52.004+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:52.043 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:52.040+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:52.955 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:52.955+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:52.983 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:52.983+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:53.062 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:53.061+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:53.914 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:53.914+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:53.950 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:53.950+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:54.048 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:54.048+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:54.894 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:54.894+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:54.997 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:54.997+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:55.025 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:55.025+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:55.885 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:55.885+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:55.952 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:55.952+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:56.074 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:56.074+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:56.879 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:56.879+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:56.916 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:56.915+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:57.035 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:57.034+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:57.926 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:57.926+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:57.929 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:57.929+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:57.998 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:57.998+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:58.940 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:58.940+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:58.960 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:58.960+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:59.042 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:58:59.042+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:59.926 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:58:59.925+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:58:59.961 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:58:59.961+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:00.017 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:00.017+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:00.926 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:00.926+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:00.987 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:00.987+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:00.999 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:00.999+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:01.901 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:01.900+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:01.965 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:01.965+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:01.972 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:01.972+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:02.928 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:02.928+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:02.955 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:02.955+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:03.020 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:03.018+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:03.929 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:03.929+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:03.975 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:03.975+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:04.007 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:04.007+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:04.887 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:04.886+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:04.946 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:04.945+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:04.987 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:04.987+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:05.926 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:05.926+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:05.954 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:05.954+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:05.986 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:05.986+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:06.920 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:06.920+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:06.945 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:06.945+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:06.946 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:06.945+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:07.911 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:07.911+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:07.940 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:07.940+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:07.987 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:07.986+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:08.926 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:08.925+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:08.942 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:08.942+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:08.985 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:08.985+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:09.881 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:09.880+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:09.926 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:09.926+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:09.950 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:09.949+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:10.847 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:10.847+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:10.910 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:10.911+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:10.970 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:10.970+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:11.836 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:11.836+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:11.934 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:11.934+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:12.008 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:12.009+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:12.850 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:12.851+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:12.980 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:12.981+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:13.053 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:13.054+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:13.838 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:13.839+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:13.981 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:13.982+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:14.070 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:14.070+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:14.831 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:14.832+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:15.009 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:15.010+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:15.044 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:15.045+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:15.838 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:15.839+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:16.025 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:16.025+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:16.058 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:16.059+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:16.850 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:16.851+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:17.034 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:17.034+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:17.069 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:17.070+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:17.841 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:17.842+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:18.018 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:18.019+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:18.027 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:18.028+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:18.803 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:18.804+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:19.020 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:19.021+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:19.046 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:19.047+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:19.756 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:19.757+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:19.999 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:20.000+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:20.011 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:20.012+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:20.805 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:20.806+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:20.978 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:20.979+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:21.006 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:21.007+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:21.854 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:21.855+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:22.011 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:22.012+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:22.017 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:22.018+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:22.850 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:22.850+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:23.016 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:23.016+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:23.027 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:23.028+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:23.861 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:23.859+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:23.994 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:23.995+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:24.068 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:24.068+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:24.855 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:24.856+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:25.016 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:25.016+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:25.046 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:25.045+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:25.905 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:25.904+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:26.006 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:26.006+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:26.072 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:26.072+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:26.875 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:26.875+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:27.019 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:27.019+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:27.040 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:27.040+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:27.915 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:27.915+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:28.016 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:28.016+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:28.060 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:28.059+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:28.964 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:28.963+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:29.001 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:29.001+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:29.036 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:29.036+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:29.940 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:29.939+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:29.951 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:29.951+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:30.026 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:30.025+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:30.931 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:30.931+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:30.946 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:30.946+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:31.049 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:31.049+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:31.916 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:31.916+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:31.943 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:31.943+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:32.085 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:32.085+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:32.917 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:32.917+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:32.939 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:32.939+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:33.055 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:33.055+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:33.922 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:33.922+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:33.933 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:33.933+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:34.025 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:34.025+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:34.895 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:34.894+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:34.918 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:34.918+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:35.061 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:35.061+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:35.864 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:35.864+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:35.913 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:35.913+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:36.042 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:36.042+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:36.827 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:36.826+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:36.920 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:36.919+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:37.057 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:37.057+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:37.869 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:37.869+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:37.950 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:37.950+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:38.045 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:38.045+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:38.848 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:38.847+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:38.969 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:38.968+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:39.082 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:39.081+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:39.889 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:39.889+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:39.994 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:39.993+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:40.064 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:40.064+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:40.876 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:40.875+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:41.016 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:41.016+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:41.081 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:41.081+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:41.836 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:41.835+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:42.006 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:42.006+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:42.131 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:42.131+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:42.880 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:42.880+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:43.034 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:43.034+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:43.098 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:43.098+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:43.910 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:43.910+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:44.057 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:44.056+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:44.105 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:44.104+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:44.907 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:44.907+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:45.040 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:45.039+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:45.125 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:45.125+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:45.877 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:45.876+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:46.043 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:46.043+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:46.145 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:46.145+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:46.912 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:46.912+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:47.038 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:47.038+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:47.191 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:47.191+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:47.873 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:47.873+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:48.061 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:48.061+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:48.195 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:48.194+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:48.880 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:48.880+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:49.102 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:49.101+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:49.223 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:49.223+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:49.928 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:49.928+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:50.110 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:50.110+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:50.217 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:50.216+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:50.960 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:50.959+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:51.063 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:51.063+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:51.246 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:51.245+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:51.926 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:51.926+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:52.030 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:52.030+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:52.259 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:52.259+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:52.891 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:52.891+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:53.038 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:53.038+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:53.278 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:53.278+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:53.857 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:53.857+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:54.039 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:54.038+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:54.270 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:54.270+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:54.868 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:54.868+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:55.087 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:55.087+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:55.283 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:55.283+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:55.830 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:55.830+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:56.040 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:56.041+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:56.250 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:56.250+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:56.841 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:56.842+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:56.991 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:56.992+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:57.252 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:57.253+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:57.846 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:57.847+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:57.978 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:57.979+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:58.223 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:58.224+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:58.818 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:58.819+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:58.944 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:58.945+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:59.225 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T18:59:59.226+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:59.788 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T18:59:59.789+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T18:59:59.938 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T18:59:59.939+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:00.212 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:00.213+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:00.787 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:00.788+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:00.921 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:00.921+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:01.171 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:01.172+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:01.824 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:01.825+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:01.970 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:01.971+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:02.170 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:02.171+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:02.817 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:02.818+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:03.019 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:03.020+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:03.124 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:03.125+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:03.834 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:03.835+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:04.000 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:04.001+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:04.083 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:04.084+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:04.826 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:04.827+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:04.980 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:04.980+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:05.087 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:05.088+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:05.809 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:05.809+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:05.968 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:05.969+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:06.094 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:06.095+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:06.775 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:06.776+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:06.996 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:06.996+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:07.113 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:07.114+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:07.766 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:07.767+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:07.981 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:07.982+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:08.151 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:08.151+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:08.768 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:08.768+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:08.945 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:08.946+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:09.161 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:09.161+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:09.779 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:09.779+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:09.899 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:09.899+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:10.113 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:10.113+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:10.749 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:10.749+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:10.913 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:10.912+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:11.158 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:11.157+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:11.765 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:11.765+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:11.941 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:11.939+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:12.192 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:12.192+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:12.788 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:12.788+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:12.956 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:12.956+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:13.199 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:13.199+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:13.755 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:13.755+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:13.962 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:13.962+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:14.225 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:14.225+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:14.735 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:14.735+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:14.939 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:14.938+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:15.229 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:15.229+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:15.687 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:15.686+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:15.896 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:15.896+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:16.277 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:16.277+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:16.639 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:16.639+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:16.926 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:16.926+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:17.273 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:17.273+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:17.596 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:17.596+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:17.936 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:17.935+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:18.231 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:18.231+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:18.589 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:18.589+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:18.890 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:18.887+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:19.247 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:19.247+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:19.631 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:19.631+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:19.853 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:19.852+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:20.264 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:20.264+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:20.589 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:20.589+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:20.833 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:20.832+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:21.222 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:21.222+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:21.633 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:21.633+0000 7f6da0215700 -1 osd.0 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:21.865 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:21.865+0000 7f621ffaa700 -1 osd.2 160 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:22.265 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:22.264+0000 7f7f6d9cc700 -1 osd.1 160 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:22.658 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:22.657+0000 7f6da0215700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:22.888 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:22.888+0000 7f621ffaa700 -1 osd.2 161 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:23.266 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:23.266+0000 7f7f6d9cc700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:23.619 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:23.619+0000 7f6da0215700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:23.881 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:23.880+0000 7f621ffaa700 -1 osd.2 161 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:24.225 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:24.225+0000 7f7f6d9cc700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:24.600 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:24.599+0000 7f6da0215700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:24.877 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:24.877+0000 7f621ffaa700 -1 osd.2 161 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:25.185 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:25.185+0000 7f7f6d9cc700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:25.606 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:25.606+0000 7f6da0215700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:25.860 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:25.860+0000 7f621ffaa700 -1 osd.2 161 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:26.164 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:26.164+0000 7f7f6d9cc700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:26.621 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:26.620+0000 7f6da0215700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:26.837 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:26.837+0000 7f621ffaa700 -1 osd.2 161 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:27.141 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:27.141+0000 7f7f6d9cc700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:27.655 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:27.655+0000 7f6da0215700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:27.809 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:27.809+0000 7f621ffaa700 -1 osd.2 161 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:28.116 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:28.116+0000 7f7f6d9cc700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:28.630 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:28.630+0000 7f6da0215700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:28.775 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:28.775+0000 7f621ffaa700 -1 osd.2 161 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:29.088 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:29.088+0000 7f7f6d9cc700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:29.633 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:29.633+0000 7f6da0215700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:29.770 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:29.770+0000 7f621ffaa700 -1 osd.2 161 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:30.077 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:30.077+0000 7f7f6d9cc700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:30.590 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:30.590+0000 7f6da0215700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:30.722 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:30.722+0000 7f621ffaa700 -1 osd.2 161 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:31.121 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:31.121+0000 7f7f6d9cc700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:31.611 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:31.611+0000 7f6da0215700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:31.739 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:31.739+0000 7f621ffaa700 -1 osd.2 161 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:32.126 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:32.125+0000 7f7f6d9cc700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:32.609 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:32.609+0000 7f6da0215700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:32.704 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:32.704+0000 7f621ffaa700 -1 osd.2 161 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:33.174 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:33.173+0000 7f7f6d9cc700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:33.578 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:33.578+0000 7f6da0215700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:33.754 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:33.753+0000 7f621ffaa700 -1 osd.2 161 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:34.166 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:34.166+0000 7f7f6d9cc700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:34.613 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:34.613+0000 7f6da0215700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:34.751 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:34.751+0000 7f621ffaa700 -1 osd.2 161 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:35.209 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:35.208+0000 7f7f6d9cc700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:35.611 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:35.611+0000 7f6da0215700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:35.767 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:35.767+0000 7f621ffaa700 -1 osd.2 161 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:36.238 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:36.237+0000 7f7f6d9cc700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:36.603 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:36.603+0000 7f6da0215700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:36.729 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:36.728+0000 7f621ffaa700 -1 osd.2 161 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:37.256 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:37.256+0000 7f7f6d9cc700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:37.600 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:37.600+0000 7f6da0215700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:37.742 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:37.741+0000 7f621ffaa700 -1 osd.2 161 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:38.247 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:38.247+0000 7f7f6d9cc700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:38.609 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:38.609+0000 7f6da0215700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:38.746 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:38.747+0000 7f621ffaa700 -1 osd.2 161 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:39.254 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:39.254+0000 7f7f6d9cc700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:39.659 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:39.659+0000 7f6da0215700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:39.789 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:39.789+0000 7f621ffaa700 -1 osd.2 161 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:40.259 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:40.259+0000 7f7f6d9cc700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:40.633 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:40.633+0000 7f6da0215700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:40.763 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:40.763+0000 7f621ffaa700 -1 osd.2 161 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:41.266 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:41.267+0000 7f7f6d9cc700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:41.629 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:41.629+0000 7f6da0215700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:41.776 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:41.777+0000 7f621ffaa700 -1 osd.2 161 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:42.220 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:42.221+0000 7f7f6d9cc700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:42.591 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:42.592+0000 7f6da0215700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:42.784 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:42.785+0000 7f621ffaa700 -1 osd.2 161 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:43.196 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:43.197+0000 7f7f6d9cc700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:43.560 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:43.561+0000 7f6da0215700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:43.807 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:43.808+0000 7f621ffaa700 -1 osd.2 161 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:44.245 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:44.246+0000 7f7f6d9cc700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:44.532 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:44.532+0000 7f6da0215700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:44.796 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:44.797+0000 7f621ffaa700 -1 osd.2 161 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:45.258 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:45.259+0000 7f7f6d9cc700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:45.502 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:45.503+0000 7f6da0215700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:45.833 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:45.834+0000 7f621ffaa700 -1 osd.2 161 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:46.298 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:46.299+0000 7f7f6d9cc700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:46.543 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:46.543+0000 7f6da0215700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:46.850 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:46.850+0000 7f621ffaa700 -1 osd.2 161 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:47.324 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:47.325+0000 7f7f6d9cc700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:47.580 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:47.580+0000 7f6da0215700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:47.871 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:47.872+0000 7f621ffaa700 -1 osd.2 161 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:48.358 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:48.359+0000 7f7f6d9cc700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:48.581 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:48.579+0000 7f6da0215700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:48.910 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:48.911+0000 7f621ffaa700 -1 osd.2 161 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:49.389 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:49.390+0000 7f7f6d9cc700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:49.582 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:49.583+0000 7f6da0215700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:49.956 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:49.957+0000 7f621ffaa700 -1 osd.2 161 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:50.342 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:50.343+0000 7f7f6d9cc700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:50.578 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:50.579+0000 7f6da0215700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:50.959 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:50.960+0000 7f621ffaa700 -1 osd.2 161 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:51.314 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:51.315+0000 7f7f6d9cc700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:51.551 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:51.552+0000 7f6da0215700 -1 osd.0 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:51.932 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:51.932+0000 7f621ffaa700 -1 osd.2 161 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:52.270 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:52.270+0000 7f7f6d9cc700 -1 osd.1 161 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:52.585 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:52.585+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:52.977 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:52.978+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:53.277 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:53.277+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:53.590 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:53.590+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:54.025 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:54.025+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:54.306 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:54.306+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:54.568 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:54.568+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:54.983 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:54.982+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:55.320 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:55.320+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:55.587 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:55.586+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:55.936 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:55.936+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:56.290 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:56.289+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:56.583 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:56.583+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:56.899 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:56.898+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:57.294 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:57.294+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:57.587 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:57.587+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:57.946 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:57.945+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:58.334 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:58.333+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:58.566 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:58.566+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:58.970 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:58.970+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:59.319 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:00:59.319+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:59.570 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:00:59.570+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:00:59.976 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:00:59.976+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:00.352 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:00.352+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:00.606 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:00.606+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:01.016 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:01.016+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:01.377 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:01.376+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:01.590 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:01.589+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:02.028 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:02.027+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:02.328 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:02.328+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:02.631 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:02.631+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:02.992 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:02.991+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:03.336 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:03.336+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:03.666 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:03.666+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:04.041 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:04.040+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:04.306 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:04.306+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:04.679 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:04.679+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:05.077 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:05.076+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:05.339 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:05.338+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:05.651 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:05.650+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:06.115 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:06.115+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:06.301 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:06.300+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:06.620 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:06.620+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:07.160 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:07.160+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:07.304 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:07.304+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:07.615 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:07.614+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:08.169 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:08.169+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:08.285 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:08.285+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:08.626 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:08.626+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:09.211 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:09.211+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:09.334 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:09.334+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:09.585 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:09.584+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:10.206 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:10.206+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:10.330 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:10.330+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:10.634 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:10.633+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:11.209 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:11.209+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:11.334 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:11.333+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:11.642 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:11.642+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:12.232 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:12.232+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:12.370 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:12.370+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:12.618 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:12.618+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:13.244 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:13.244+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:13.387 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:13.387+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:13.652 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:13.652+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:14.204 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:14.203+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:14.370 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:14.370+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:14.697 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:14.696+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:15.197 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:15.197+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:15.412 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:15.412+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:15.726 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:15.726+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:16.212 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:16.212+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:16.386 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:16.386+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:16.775 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:16.775+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:17.210 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:17.210+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:17.362 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:17.361+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:17.748 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:17.748+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:18.163 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:18.163+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:18.383 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:18.383+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:18.792 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:18.792+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:19.205 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:19.205+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:19.366 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:19.365+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:19.781 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:19.781+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:20.208 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:20.208+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:20.342 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:20.342+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:20.822 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:20.822+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:21.229 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:21.229+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:21.373 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:21.373+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:21.782 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:21.782+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:22.201 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:22.201+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:22.343 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:22.342+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:22.811 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:22.810+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:23.250 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:23.249+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:23.386 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:23.386+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:23.790 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:23.789+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:24.279 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:24.279+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:24.401 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:24.399+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:24.780 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:24.780+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:25.248 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:25.248+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:25.434 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:25.433+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:25.750 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:25.751+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:26.202 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:26.203+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:26.414 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:26.414+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:26.765 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:26.766+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:27.199 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:27.200+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:27.376 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:27.377+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:27.764 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:27.765+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:28.168 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:28.168+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:28.399 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:28.400+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:28.779 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:28.780+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:29.128 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:29.129+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:29.376 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:29.377+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:29.772 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:29.772+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:30.127 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:30.128+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:30.367 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:30.368+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:30.761 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:30.761+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:31.130 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:31.131+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:31.349 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:31.350+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:31.779 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:31.780+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:32.174 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:32.175+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:32.400 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:32.401+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:32.731 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:32.732+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:33.190 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:33.191+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:33.392 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:33.392+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:33.714 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:33.715+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:34.194 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:34.194+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:34.353 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:34.354+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:34.755 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:34.756+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:35.209 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:35.210+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:35.364 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:35.365+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:35.751 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:35.752+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:36.212 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:36.213+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:36.369 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:36.370+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:36.742 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:36.742+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:37.207 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:37.207+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:37.352 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:37.352+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:37.748 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:37.748+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:38.209 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:38.210+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:38.345 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:38.343+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:38.773 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:38.773+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:39.215 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:39.213+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:39.365 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:39.363+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:39.737 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:39.737+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:40.203 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:40.203+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:40.385 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:40.385+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:40.755 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:40.754+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:41.204 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:41.204+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:41.358 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:41.358+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:41.729 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:41.729+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:42.176 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:42.175+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:42.340 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:42.340+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:42.711 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:42.711+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:43.196 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:43.196+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:43.325 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:43.325+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:43.735 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:43.735+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:44.200 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:44.200+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:44.321 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:44.321+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:44.714 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:44.713+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:45.224 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:45.224+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:45.335 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:45.335+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:45.738 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:45.738+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:46.218 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:46.218+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:46.381 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:46.381+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:46.735 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:46.734+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:47.266 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:47.266+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:47.359 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:47.359+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:47.763 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:47.763+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:48.290 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:48.290+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:48.359 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:48.359+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:48.798 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:48.797+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:49.304 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:49.304+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:49.401 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:49.401+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:49.759 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:49.759+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:50.314 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:50.313+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:50.451 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:50.451+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:50.753 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:50.753+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:51.351 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:51.351+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:51.403 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:51.403+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:51.753 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:51.753+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:52.336 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:52.336+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:52.397 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:52.397+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:52.707 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:52.707+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:53.327 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:53.327+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:53.445 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:53.445+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:53.667 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:53.666+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:54.367 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:54.367+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:54.466 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:54.465+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:54.711 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:54.711+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:55.386 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:55.386+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:55.487 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:55.487+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:55.666 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:55.665+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:56.364 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:56.364+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:56.464 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:56.464+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:56.641 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:56.641+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:57.321 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:57.320+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:57.483 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:57.483+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:57.615 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:57.615+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:58.360 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:58.360+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:58.436 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:58.436+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:58.626 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:58.625+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:59.378 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:01:59.378+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:59.452 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:01:59.451+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:01:59.618 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:01:59.618+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:00.400 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:00.399+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:00.453 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:00.453+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:00.627 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:00.627+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:01.414 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:01.414+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:01.443 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:01.443+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:01.666 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:01.666+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:02.395 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:02.394+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:02.486 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:02.486+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:02.659 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:02.658+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:03.360 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:03.359+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:03.444 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:03.443+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:03.658 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:03.658+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:04.348 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:04.348+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:04.453 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:04.452+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:04.613 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:04.612+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:05.338 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:05.338+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:05.471 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:05.471+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:05.589 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:05.589+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:06.373 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:06.372+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:06.451 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:06.451+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:06.618 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:06.618+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:07.323 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:07.323+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:07.493 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:07.493+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:07.664 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:07.664+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:08.315 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:08.315+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:08.523 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:08.523+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:08.698 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:08.698+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:09.277 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:09.277+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:09.529 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:09.529+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:09.682 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:09.682+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:10.272 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:10.272+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:10.486 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:10.486+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:10.731 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:10.732+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:11.308 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:11.309+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:11.537 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:11.537+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:11.766 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:11.767+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:12.332 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:12.333+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:12.521 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:12.522+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:12.746 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:12.747+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:13.333 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:13.334+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:13.474 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:13.475+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:13.769 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:13.769+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:14.301 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:14.302+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:14.473 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:14.474+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:14.738 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:14.739+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:15.297 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:15.297+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:15.482 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:15.482+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:15.740 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:15.741+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:16.305 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:16.306+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:16.460 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:16.461+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:16.760 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:16.761+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:17.291 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:17.292+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:17.478 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:17.479+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:17.759 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:17.759+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:18.302 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:18.302+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:18.519 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:18.519+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:18.789 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:18.790+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:19.271 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:19.272+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:19.505 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:19.505+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:19.758 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:19.759+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:20.319 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:20.320+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:20.496 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:20.497+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:20.794 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:20.795+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:21.343 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:21.343+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:21.464 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:21.465+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:21.793 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:21.793+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:22.393 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:22.393+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:22.510 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:22.511+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:22.782 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:22.782+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:23.421 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:23.420+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:23.533 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:23.533+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:23.777 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:23.776+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:24.397 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:24.397+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:24.562 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:24.561+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:24.768 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:24.768+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:25.355 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:25.355+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:25.531 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:25.531+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:25.800 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:25.800+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:26.333 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:26.333+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:26.532 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:26.532+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:26.772 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:26.772+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:27.339 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:27.338+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:27.574 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:27.573+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:27.748 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:27.748+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:28.289 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:28.289+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:28.598 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:28.598+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:28.716 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:28.716+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:29.339 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:29.339+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:29.636 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:29.636+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:29.684 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:29.684+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:30.375 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:30.375+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:30.649 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:30.649+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:30.655 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:30.655+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:31.344 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:31.343+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:31.637 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:31.637+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:31.655 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:31.655+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:32.326 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:32.326+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:32.663 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:32.662+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:32.695 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:32.695+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:33.293 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:33.292+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:33.669 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:33.668+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:33.674 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:33.674+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:34.286 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:34.286+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:34.682 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:34.682+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:34.703 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:34.702+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:35.291 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:35.291+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:35.712 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:35.711+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:35.727 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:35.726+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:36.337 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:36.337+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:36.747 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:36.747+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:36.756 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:36.756+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:37.382 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:37.382+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:37.724 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:37.724+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:37.780 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:37.779+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:38.411 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:38.410+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:38.717 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:38.717+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:38.743 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:38.743+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:39.371 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:39.371+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:39.720 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:39.720+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:39.725 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:39.725+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:40.401 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:40.401+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:40.716 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:40.716+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:40.764 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:40.764+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:41.432 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:41.432+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 6 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:41.722 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:41.722+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:41.791 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:41.790+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:42.481 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:42.478+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 5 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:42.735 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:42.735+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:42.833 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:42.833+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:43.464 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:43.464+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 5 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:43.705 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:43.705+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:43.877 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:43.877+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:44.444 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:44.443+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 5 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:44.733 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:44.730+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:44.850 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:44.850+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:45.435 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:45.435+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 5 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:45.754 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:45.754+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:45.862 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:45.862+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:46.463 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:46.463+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 5 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:46.740 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:46.740+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:46.843 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:46.843+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:47.440 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:47.440+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 5 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:47.728 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:47.728+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:47.823 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:47.823+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:48.458 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:48.458+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 5 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:48.689 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:48.688+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:48.833 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:48.831+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:49.424 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:49.424+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 5 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:49.678 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:49.677+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:49.830 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:49.830+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:50.401 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:50.401+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 5 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:50.716 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:50.716+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:50.810 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:50.810+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:51.419 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:51.419+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 5 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:51.677 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:51.677+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:51.838 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:51.838+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:52.380 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:52.380+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 5 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:52.637 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:52.636+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:52.831 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:52.831+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:53.394 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:53.394+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 5 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:53.634 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:53.634+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:53.834 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:53.833+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:54.426 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:54.425+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 5 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:54.667 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:54.666+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:54.790 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:54.791+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:55.398 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:55.399+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 5 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:55.676 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:55.677+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:55.766 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:55.767+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:56.362 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:56.363+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 5 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:56.652 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:56.653+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:56.776 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:56.777+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:57.401 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:57.402+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 5 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:57.661 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:57.662+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:57.771 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:57.771+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:58.415 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:58.416+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 5 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:58.648 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:58.649+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:58.728 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:58.728+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:59.452 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:02:59.452+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 5 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:59.673 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:02:59.674+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:02:59.693 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:02:59.694+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:00.421 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:00.422+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 5 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:00.669 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:00.670+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:00.680 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:00.681+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:01.467 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:01.467+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 5 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:01.644 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:01.645+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:01.684 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:01.685+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:02.487 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:02.487+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 5 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:02.639 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:02.640+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:02.686 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:02.687+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:03.449 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:03.450+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 5 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:03.633 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:03.634+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:03.728 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:03.729+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:04.473 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:04.474+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 5 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:04.639 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:04.640+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:04.681 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:04.681+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:05.442 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:05.443+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 5 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:05.628 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:05.629+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:05.645 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:05.646+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:06.427 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:06.428+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 5 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:06.627 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:06.628+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:06.662 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:06.663+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:07.419 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:07.420+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 5 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:07.634 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:07.635+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:07.712 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:07.712+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:08.427 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:08.427+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 5 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:08.646 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:08.645+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:08.695 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:08.695+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:09.434 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:09.433+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 5 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:09.649 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:09.649+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:09.707 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:09.707+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:10.433 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:10.433+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 5 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:10.610 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:10.610+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:10.711 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:10.711+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:11.384 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:11.384+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 5 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:11.659 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:11.659+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:11.679 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:11.679+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:12.419 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:12.419+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:12.672 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:12.672+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:12.674 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:12.674+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:13.416 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:13.416+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:13.682 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:13.681+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:13.696 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:13.696+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:14.389 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:14.389+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:14.726 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:14.725+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:14.736 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:14.736+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:15.436 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:15.435+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:15.702 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:15.701+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:15.755 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:15.755+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:16.464 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:16.464+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:16.736 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:16.736+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:16.775 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:16.775+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:17.468 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:17.468+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:17.778 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:17.778+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:17.802 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:17.801+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:18.442 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:18.441+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:18.735 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:18.734+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:18.781 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:18.781+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:19.397 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:19.397+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:19.701 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:19.701+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:19.810 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:19.810+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:20.353 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:20.353+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:20.666 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:20.666+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:20.817 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:20.816+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:21.398 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:21.398+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:21.628 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:21.628+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:21.807 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:21.807+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:22.431 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:22.430+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:22.624 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:22.623+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:22.853 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:22.853+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:23.439 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:23.439+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:23.628 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:23.628+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:23.887 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:23.887+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:24.423 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:24.422+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:24.579 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:24.579+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:24.861 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:24.861+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:25.415 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:25.415+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:25.562 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:25.562+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:25.818 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:25.818+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:26.455 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:26.455+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:26.587 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:26.587+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:26.834 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:26.834+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:27.425 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:27.424+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:27.625 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:27.624+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:27.798 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:27.798+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:28.446 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:28.446+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:28.632 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:28.632+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:28.802 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:28.802+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:29.467 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:29.467+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:29.598 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:29.597+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:29.836 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:29.836+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:30.459 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:30.458+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:30.628 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:30.628+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:30.854 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:30.854+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:31.455 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:31.454+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:31.580 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:31.579+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:31.854 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:31.854+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:32.431 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:32.431+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:32.555 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:32.555+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:32.858 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:32.858+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:33.436 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:33.435+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:33.582 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:33.582+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:33.835 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:33.834+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:34.483 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:34.483+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:34.577 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:34.577+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:34.821 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:34.821+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:35.496 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:35.495+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:35.610 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:35.609+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:35.837 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:35.837+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:36.493 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:36.493+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:36.601 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:36.601+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:36.840 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:36.840+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:37.492 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:37.492+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:37.614 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:37.614+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:37.835 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:37.835+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:38.526 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:38.526+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:38.607 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:38.606+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:38.872 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:38.872+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:39.526 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:39.527+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:39.635 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:39.636+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:39.826 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:39.826+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:40.529 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:40.530+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:40.616 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:40.616+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:40.877 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:40.877+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:41.565 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:41.566+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:41.611 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:41.612+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:41.903 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:41.904+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:42.575 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:42.575+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:42.595 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:42.596+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:42.923 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:42.924+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:43.599 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:43.600+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:43.614 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:43.615+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:43.953 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:43.954+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:44.568 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:44.569+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:44.643 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:44.644+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:44.972 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:44.973+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:45.588 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:45.589+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:45.685 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:45.686+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:45.926 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:45.927+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:46.587 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:46.588+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:46.661 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:46.662+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:46.923 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:46.924+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:47.630 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:47.631+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:47.663 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:47.664+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:47.951 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:47.952+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:48.669 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:48.669+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:48.695 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:48.696+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:48.929 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:48.930+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:49.632 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:49.632+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:49.689 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:49.690+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:49.909 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:49.909+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:50.660 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:50.660+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:50.675 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:50.676+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:50.871 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:50.872+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:51.642 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:51.643+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:51.706 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:51.706+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:51.837 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:51.837+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:52.594 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:52.594+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:52.728 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:52.728+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:52.870 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:52.870+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:53.631 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:53.631+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:53.709 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:53.709+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:53.863 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:53.863+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:54.588 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:54.588+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:54.723 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:54.722+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:54.867 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:54.867+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:55.612 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:55.612+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:55.743 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:55.743+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:55.838 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:55.838+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:56.610 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:56.610+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:56.765 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:56.765+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:56.794 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:56.794+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:57.601 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:57.601+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:57.751 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:57.751+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:57.765 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:57.765+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:58.581 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:58.581+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:58.739 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:58.739+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:58.759 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:58.759+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:59.560 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:03:59.559+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:59.705 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:03:59.704+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:03:59.772 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:03:59.772+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:00.535 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:00.534+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:00.727 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:00.727+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:00.728 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:00.728+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:01.564 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:01.564+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:01.722 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:01.722+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:01.766 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:01.765+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:02.613 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:02.612+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:02.727 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:02.727+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:02.757 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:02.757+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:03.649 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:03.649+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:03.757 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:03.757+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:03.799 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:03.799+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:04.662 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:04.661+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:04.732 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:04.732+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:04.797 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:04.797+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:05.630 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:05.630+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:05.743 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:05.743+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:05.790 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:05.790+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:06.624 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:06.624+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:06.778 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:06.778+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:06.783 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:06.783+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:07.670 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:07.670+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:07.805 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:07.805+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:07.819 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:07.819+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:08.712 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:08.712+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:08.809 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:08.809+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:08.836 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:08.836+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:09.668 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:09.667+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:09.786 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:09.785+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:09.789 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:09.789+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:10.708 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:10.708+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:10.747 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:10.747+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:10.799 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:10.799+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:11.756 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:11.756+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:11.787 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:11.787+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:11.797 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:11.797+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:12.711 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:12.711+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:12.754 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:12.754+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:12.802 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:12.801+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:13.723 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:13.722+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:13.748 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:13.748+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:13.835 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:13.834+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:14.746 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:14.746+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:14.770 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:14.770+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:14.818 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:14.817+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:15.739 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:15.738+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:15.772 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:15.772+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:15.818 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:15.818+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:16.718 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:16.718+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:16.748 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:16.748+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:16.823 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:16.822+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:17.747 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:17.747+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:17.791 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:17.791+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:17.795 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:17.795+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:18.735 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:18.735+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:18.772 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:18.772+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:18.806 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:18.806+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:19.718 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:19.718+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:19.747 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:19.747+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:19.797 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:19.797+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:20.765 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:20.765+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:20.784 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:20.784+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:20.824 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:20.824+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:21.723 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:21.724+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:21.746 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:21.746+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:21.863 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:21.863+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:22.743 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:22.742+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:22.795 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:22.795+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:22.889 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:22.889+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:23.771 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:23.772+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:23.798 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:23.799+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:23.895 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:23.896+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:24.770 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:24.771+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:24.836 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:24.837+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:24.888 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:24.889+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:25.786 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:25.787+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:25.859 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:25.860+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:25.890 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:25.891+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:26.765 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:26.766+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:26.873 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:26.874+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:26.879 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:26.880+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:27.769 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:27.770+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:27.868 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:27.869+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:27.930 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:27.931+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:28.724 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:28.725+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:28.908 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:28.908+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:28.960 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:28.961+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:29.715 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:29.716+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:29.940 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:29.941+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:29.952 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:29.953+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:30.669 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:30.670+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:30.918 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:30.918+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:30.955 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:30.956+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:31.675 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:31.675+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:31.953 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:31.954+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:31.957 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:31.958+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:32.691 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:32.691+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:32.965 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:32.965+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:32.990 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:32.991+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:33.671 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:33.672+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:33.966 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:33.967+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:33.987 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:33.988+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:34.696 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:34.697+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:34.998 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:34.999+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:35.037 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:35.038+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:35.653 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:35.653+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:35.950 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:35.951+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:35.989 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:35.990+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:36.663 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:36.663+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:36.926 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:36.926+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:36.948 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:36.948+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:37.677 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:37.677+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:37.946 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:37.946+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:37.972 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:37.972+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:38.694 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:38.693+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:38.918 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:38.918+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:39.009 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:39.008+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:39.717 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:39.717+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:39.942 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:39.942+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:40.048 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:40.048+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:40.715 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:40.715+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:40.967 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:40.967+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:41.004 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:41.004+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:41.756 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:41.756+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:41.948 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:41.948+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:42.013 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:42.013+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:42.768 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:42.768+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:42.948 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:42.948+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:42.978 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:42.978+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:43.782 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:43.781+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:43.949 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:43.949+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:44.014 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:44.014+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:44.790 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:44.790+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:44.971 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:44.971+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:45.062 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:45.062+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:45.765 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:45.764+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:45.955 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:45.955+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:46.034 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:46.034+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:46.753 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:46.753+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:46.924 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:46.923+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:47.052 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:47.052+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:47.774 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:47.774+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:47.963 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:47.963+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:48.024 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:48.024+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:48.813 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:48.812+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:48.949 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:48.949+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:49.073 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:49.073+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:49.855 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:49.854+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:49.914 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:49.914+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:50.112 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:50.112+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:50.850 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:50.850+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:50.932 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:50.932+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:51.104 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:51.103+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:51.815 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:51.815+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:51.893 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:51.892+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:52.072 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:52.072+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:52.781 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:52.781+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:52.877 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:52.877+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:53.087 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:53.087+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:53.812 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:53.811+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:53.840 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:53.839+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:54.115 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:54.115+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:54.781 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:54.781+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:54.803 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:54.803+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:55.081 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:55.081+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:55.788 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:55.787+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:55.823 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:55.823+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:56.037 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:56.037+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:56.769 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:56.769+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:56.836 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:56.834+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:57.050 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:57.050+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:57.770 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:57.770+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:57.824 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:57.823+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:58.017 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:58.017+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:58.729 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:58.729+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:58.870 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:58.870+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:59.050 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:04:59.049+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:59.753 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:04:59.753+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:04:59.839 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:04:59.838+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:00.088 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:00.088+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:00.724 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:00.724+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:00.863 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:00.863+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:01.112 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:01.112+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:01.735 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:01.735+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:01.820 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:01.820+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:02.131 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:02.131+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:02.706 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:02.706+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:02.835 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:02.835+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:03.129 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:03.129+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:03.690 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:03.689+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:03.845 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:03.844+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:04.151 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:04.151+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:04.727 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:04.727+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:04.880 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:04.880+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:05.116 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:05.116+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:05.717 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:05.717+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:05.879 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:05.880+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:06.106 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:06.106+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:06.744 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:06.744+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:06.875 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:06.875+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:07.062 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:07.062+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:07.794 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:07.795+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:07.923 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:07.922+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:08.064 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:08.064+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:08.816 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:08.817+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:08.903 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:08.903+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:09.083 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:09.084+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:09.850 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:09.851+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:09.879 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:09.879+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:10.086 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:10.087+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:10.803 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:10.804+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:10.906 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:10.907+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:11.070 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:11.071+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:11.782 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:11.783+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:11.945 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:11.946+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:12.072 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:12.073+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:12.788 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:12.789+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:12.967 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:12.967+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:13.060 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:13.061+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:13.753 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:13.754+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:13.974 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:13.974+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:14.022 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:14.022+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:14.800 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:14.801+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:14.965 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:14.966+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:14.984 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:14.985+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:15.790 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:15.791+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:15.986 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:15.986+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:15.990 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:15.991+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:16.748 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:16.749+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:16.963 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:16.964+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:17.039 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:17.040+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:17.700 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:17.700+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:17.954 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:17.955+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:18.087 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:18.088+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:18.708 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:18.708+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:18.926 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:18.927+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:19.059 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:19.059+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:19.727 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:19.728+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:19.963 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:19.964+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:20.071 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:20.072+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:20.776 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:20.777+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:20.943 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:20.944+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:21.025 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:21.026+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:21.739 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:21.739+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:21.916 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:21.916+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:22.000 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:22.000+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:22.706 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:22.705+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:22.881 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:22.881+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:22.970 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:22.970+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:23.742 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:23.741+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:23.929 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:23.929+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:23.933 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:23.933+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:24.728 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:24.728+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:24.914 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:24.913+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:24.923 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:24.923+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:25.695 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:25.694+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:25.903 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:25.903+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:25.921 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:25.920+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:26.708 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:26.708+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:26.863 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:26.862+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:26.884 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:26.884+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:27.736 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:27.736+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:27.870 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:27.869+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:27.891 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:27.891+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:28.732 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:28.732+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:28.856 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:28.856+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:28.903 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:28.902+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:29.764 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:29.763+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:29.812 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:29.812+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:29.926 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:29.926+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:30.766 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:30.766+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:30.856 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:30.856+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:30.906 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:30.906+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:31.800 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:31.799+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:31.847 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:31.847+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:31.874 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:31.874+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:32.786 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:32.786+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:32.837 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:32.837+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:32.872 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:32.872+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:33.836 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:33.836+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:33.853 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:33.853+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:33.859 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:33.859+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:34.808 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:34.808+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:34.821 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:34.821+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:34.840 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:34.840+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:35.826 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:35.825+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:35.855 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:35.854+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:35.856 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:35.856+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:36.825 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:36.825+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:36.859 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:36.859+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:36.876 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:36.876+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:37.822 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:37.822+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:37.866 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:37.866+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:37.869 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:37.869+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:38.832 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:38.832+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:38.845 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:38.845+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:38.847 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:38.847+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:39.808 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:39.808+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:39.849 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:39.848+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:39.856 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:39.856+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:40.806 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:40.806+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:40.825 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:40.825+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:40.880 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:40.880+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:41.780 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:41.780+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:41.800 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:41.800+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:41.911 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:41.911+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:42.796 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:42.795+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:42.850 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:42.816+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:42.886 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:42.886+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:43.791 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:43.791+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:43.816 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:43.816+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:43.842 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:43.842+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:44.826 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:44.826+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:44.831 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:44.831+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:44.841 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:44.841+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:45.844 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:45.843+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:45.846 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:45.846+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:45.872 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:45.872+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:46.843 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:46.842+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:46.884 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:46.884+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:46.888 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:46.888+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:47.809 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:47.809+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:47.910 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:47.910+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:47.919 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:47.918+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:48.837 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:48.834+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:48.890 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:48.890+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:48.908 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:48.908+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:49.807 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:49.807+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:49.917 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:49.917+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:49.925 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:49.925+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:50.816 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:50.815+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:50.920 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:50.920+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:50.962 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:50.962+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:51.853 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:51.854+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:51.877 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:51.877+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:51.983 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:51.984+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:52.837 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:52.837+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:52.856 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:52.856+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:52.935 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:52.935+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:53.826 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:53.827+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:53.867 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:53.868+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:53.960 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:53.960+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:54.826 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:54.827+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:54.877 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:54.878+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:54.943 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:54.943+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:55.786 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:55.787+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:55.851 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:55.852+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:55.951 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:55.952+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:56.796 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:56.797+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:56.862 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:56.863+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:56.996 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:56.997+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:57.816 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:57.817+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:57.821 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:57.822+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:57.972 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:57.973+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:58.802 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:58.802+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:58.865 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:58.866+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:59.016 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:59.017+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:59.827 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:05:59.827+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:59.870 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:05:59.871+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:05:59.978 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:05:59.979+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:00.811 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:00.812+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:00.829 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:00.830+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:00.996 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:00.997+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:01.801 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:01.802+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:01.816 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:01.817+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:02.030 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:02.031+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:02.771 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:02.772+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:02.827 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:02.828+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:03.050 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:03.051+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:03.740 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:03.740+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:03.834 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:03.834+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:04.047 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:04.048+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:04.747 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:04.748+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:04.872 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:04.873+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:05.048 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:05.049+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:05.706 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:05.707+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:05.865 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:05.865+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:06.043 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:06.044+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:06.754 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:06.753+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:06.888 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:06.888+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:07.048 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:07.048+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:07.755 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:07.755+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:07.928 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:07.928+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:08.082 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:08.082+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:08.710 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:08.710+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:08.958 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:08.957+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:09.125 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:09.125+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:09.731 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:09.731+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:09.943 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:09.942+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:10.089 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:10.088+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:10.746 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:10.746+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:10.910 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:10.910+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:11.063 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:11.063+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:11.745 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:11.745+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:11.916 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:11.916+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:12.096 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:12.096+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:12.704 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:12.704+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:12.886 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:12.885+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:13.130 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:13.129+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:13.699 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:13.699+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:13.867 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:13.867+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:14.082 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:14.082+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:14.707 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:14.707+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:14.820 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:14.820+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:15.043 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:15.042+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:15.662 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:15.661+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:15.826 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:15.826+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:16.012 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:16.012+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:16.636 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:16.636+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:16.815 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:16.814+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:17.037 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:17.037+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:17.618 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:17.618+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:17.862 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:17.862+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:18.030 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:18.030+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:18.621 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:18.621+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:18.832 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:18.832+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:18.998 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:18.998+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:19.585 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:19.585+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:19.846 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:19.846+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:19.966 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:19.966+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:20.616 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:20.615+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:20.824 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:20.824+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:21.005 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:21.005+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:21.579 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:21.579+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:21.785 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:21.785+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:21.980 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:21.980+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:22.600 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:22.599+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:22.779 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:22.779+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:23.025 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:23.025+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:23.626 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:23.626+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:23.746 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:23.745+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:24.036 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:24.036+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:24.577 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:24.576+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:24.796 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:24.796+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:25.040 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:25.040+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:25.533 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:25.532+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:25.800 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:25.800+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:26.066 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:26.066+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:26.514 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:26.514+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:26.827 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:26.827+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:27.021 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:27.021+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:27.543 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:27.543+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:27.877 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:27.876+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:28.060 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:28.059+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:28.546 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:28.546+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:28.923 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:28.923+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:29.047 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:29.047+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:29.581 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:29.581+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:29.928 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:29.928+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:30.062 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:30.062+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:30.575 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:30.575+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:30.920 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:30.919+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:31.067 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:31.066+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:31.611 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:31.610+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:31.933 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:31.933+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:32.025 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:32.025+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:32.633 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:32.633+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:32.906 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:32.906+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:33.061 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:33.060+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:33.599 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:33.598+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:33.924 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:33.924+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:34.020 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:34.019+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:34.604 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:34.604+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:34.963 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:34.963+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:34.982 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:34.982+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:35.597 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:35.597+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:35.963 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:35.962+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:36.004 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:36.004+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:36.596 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:36.597+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:36.942 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:36.941+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:36.982 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:36.982+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:37.552 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:37.552+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:37.983 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:37.983+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:38.018 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:38.017+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:38.530 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:38.531+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:39.004 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:39.005+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:39.011 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:39.012+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:39.529 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:39.530+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:40.018 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:40.018+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:40.054 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:40.055+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:40.547 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:40.548+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:41.003 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:41.004+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:41.094 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:41.094+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:41.525 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:41.526+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:42.047 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:42.047+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:42.098 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:42.099+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:42.495 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:42.495+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:43.068 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:43.069+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:43.145 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:43.143+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:43.451 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:43.451+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:44.110 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:44.110+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:44.179 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:44.180+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:44.402 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:44.403+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:45.105 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:45.106+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:45.146 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:45.147+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:45.438 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:45.438+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:46.121 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:46.122+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:46.138 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:46.139+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:46.412 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:46.412+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:47.091 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:47.092+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:47.135 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:47.136+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:47.368 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:47.368+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:48.053 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:48.054+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:48.167 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:48.167+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:48.348 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:48.349+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:49.063 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:49.064+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:49.190 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:49.190+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:49.385 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:49.385+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:50.055 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:50.056+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:50.207 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:50.208+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:50.346 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:50.346+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:51.057 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:51.058+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:51.238 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:51.239+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:51.299 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:51.300+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:52.054 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:52.053+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:52.201 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:52.201+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:52.319 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:52.319+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:53.039 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:53.039+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:53.179 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:53.179+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:53.368 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:53.368+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:54.050 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:54.049+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:54.135 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:54.135+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:54.373 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:54.373+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:55.015 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:55.015+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:55.146 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:55.146+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:55.346 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:55.345+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:56.046 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:56.046+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:56.149 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:56.149+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:56.322 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:56.321+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:57.017 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:57.017+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:57.157 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:57.157+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:57.320 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:57.320+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:58.025 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:58.025+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:58.111 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:58.111+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:58.334 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:58.334+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:58.992 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:58.991+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:59.092 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:06:59.092+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:59.364 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:06:59.363+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:06:59.946 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:06:59.945+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:00.079 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:00.079+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:00.319 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:00.319+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:00.955 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:00.954+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:01.034 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:01.033+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:01.348 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:01.348+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:01.976 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:01.975+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:01.995 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:01.995+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:02.358 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:02.358+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:02.969 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:02.969+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:03.021 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:03.021+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:03.380 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:03.380+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:03.937 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:03.937+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:04.035 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:04.035+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:04.366 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:04.366+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:04.964 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:04.964+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:05.001 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:05.001+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:05.330 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:05.330+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:05.951 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:05.951+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:05.962 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:05.961+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:06.368 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:06.367+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:06.927 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:06.927+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:07.007 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:07.007+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:07.318 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:07.318+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:07.904 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:07.904+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:08.057 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:08.057+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:08.272 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:08.272+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:08.856 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:08.855+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:09.025 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:09.025+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:09.236 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:09.236+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:09.883 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:09.883+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:10.041 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:10.041+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:10.203 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:10.203+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:10.847 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:10.846+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:11.075 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:11.075+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:11.198 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:11.198+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:11.862 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:11.862+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:12.112 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:12.112+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:12.243 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:12.243+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:12.853 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:12.852+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:13.163 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:13.162+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:13.213 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:13.213+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:13.877 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:13.876+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:14.157 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:14.157+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:14.193 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:14.193+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:14.859 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:14.858+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:15.137 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:15.137+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:15.177 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:15.177+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:15.876 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:15.875+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:16.184 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:16.183+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:16.185 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:16.185+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:16.847 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:16.846+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:17.146 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:17.145+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:17.179 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:17.179+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:17.864 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:17.863+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:18.176 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:18.176+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:18.206 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:18.205+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:18.846 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:18.846+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:19.210 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:19.210+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:19.227 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:19.227+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:19.876 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:19.875+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:20.182 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:20.182+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:20.195 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:20.195+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:20.902 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:20.902+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:21.169 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:21.169+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:21.199 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:21.199+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:21.878 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:21.878+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:22.188 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:22.188+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:22.249 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:22.248+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:22.912 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:22.912+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:23.204 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:23.204+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:23.289 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:23.289+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:23.914 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:23.915+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:24.191 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:24.192+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:24.282 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:24.283+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:24.950 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:24.951+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:25.167 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:25.168+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:25.316 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:25.316+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:25.961 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:25.962+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:26.158 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:26.159+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:26.273 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:26.274+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:26.989 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:26.989+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:27.130 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:27.130+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:27.299 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:27.300+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:28.025 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:28.026+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:28.166 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:28.167+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:28.263 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:28.264+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:29.060 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:29.061+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:29.149 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:29.150+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:29.236 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:29.237+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:30.070 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:30.071+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:30.188 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:30.189+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:30.240 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:30.241+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:31.108 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:31.109+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:31.175 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:31.176+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:31.285 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:31.286+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:32.142 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:32.143+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:32.173 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:32.174+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:32.254 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:32.255+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:33.186 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:33.186+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:33.208 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:33.209+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:33.220 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:33.221+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:34.174 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:34.174+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:34.225 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:34.226+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:34.255 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:34.256+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:35.193 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:35.193+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:35.204 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:35.205+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:35.279 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:35.280+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:36.231 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:36.232+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:36.238 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:36.238+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:36.316 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:36.317+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:37.225 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:37.224+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:37.226 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:37.226+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:37.291 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:37.290+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:38.215 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:38.215+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:38.232 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:38.232+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:38.271 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:38.271+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:39.190 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:39.190+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:39.225 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:39.225+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:39.257 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:39.257+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:40.216 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:40.216+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:40.226 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:40.226+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:40.265 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:40.265+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:41.177 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:41.177+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:41.213 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:41.213+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:41.242 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:41.242+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:42.184 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:42.183+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:42.187 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:42.187+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:42.251 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:42.251+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:43.151 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:43.151+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:43.197 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:43.197+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:43.226 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:43.226+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:44.161 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:44.161+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:44.225 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:44.225+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:44.242 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:44.242+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:45.151 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:45.151+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:45.181 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:45.181+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:45.214 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:45.214+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:46.146 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:46.146+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:46.147 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:46.147+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:46.263 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:46.263+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:47.138 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:47.138+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:47.166 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:47.165+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:47.234 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:47.234+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:48.123 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:48.123+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:48.171 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:48.171+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:48.251 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:48.251+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:49.103 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:49.103+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:49.195 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:49.194+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:49.240 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:49.240+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:50.122 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:50.122+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:50.168 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:50.168+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:50.218 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:50.218+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:51.102 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:51.101+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:51.151 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:51.151+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:51.191 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:51.191+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:52.118 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:52.118+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:52.175 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:52.175+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:52.189 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:52.189+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:53.088 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:53.087+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:53.196 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:53.196+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:53.198 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:53.198+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:54.062 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:54.062+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:54.156 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:54.156+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:54.164 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:54.164+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:55.096 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:55.095+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:55.147 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:55.147+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:55.187 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:55.186+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:56.071 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:56.071+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:56.121 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:56.120+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:56.145 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:56.145+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:57.041 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:57.041+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:57.143 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:57.143+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:57.193 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:57.193+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:58.007 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:58.007+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:58.142 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:58.141+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:58.167 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:58.167+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:58.987 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:07:58.987+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:59.121 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:07:59.121+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:07:59.192 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:07:59.191+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:00.006 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:00.006+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:00.110 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:00.110+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:00.158 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:00.157+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:01.031 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:01.031+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:01.088 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:01.088+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:01.174 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:01.174+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:02.033 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:02.033+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:02.100 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:02.100+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:02.133 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:02.133+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:03.033 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:03.033+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:03.138 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:03.138+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:03.180 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:03.180+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:04.078 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:04.077+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:04.160 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:04.159+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:04.220 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:04.220+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:05.043 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:05.042+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:05.176 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:05.176+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:05.271 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:05.271+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:06.026 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:06.026+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:06.158 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:06.158+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:06.315 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:06.314+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:06.998 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:06.998+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:07.173 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:07.173+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:07.274 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:07.274+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:07.989 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:07.989+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:08.167 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:08.168+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:08.287 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:08.288+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:09.037 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:09.038+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:09.123 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:09.124+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:09.292 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:09.293+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:10.055 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:10.056+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:10.103 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:10.104+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:10.298 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:10.299+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:11.038 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:11.039+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:11.133 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:11.134+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:11.291 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:11.292+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:12.006 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:12.007+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:12.173 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:12.174+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:12.340 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:12.341+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:12.979 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:12.980+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:13.161 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:13.162+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:13.298 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:13.298+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:14.024 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:14.025+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:14.197 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:14.197+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:14.298 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:14.299+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:14.999 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:15.000+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:15.202 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:15.203+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:15.336 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:15.337+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:15.978 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:15.979+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:16.174 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:16.175+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:16.385 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:16.386+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:16.985 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:16.986+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:17.159 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:17.159+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:17.431 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:17.431+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:18.011 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:18.011+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:18.121 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:18.122+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:18.422 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:18.423+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:19.025 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:19.026+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:19.142 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:19.143+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:19.432 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:19.432+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:20.006 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:20.006+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:20.184 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:20.185+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:20.439 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:20.440+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:21.045 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:21.046+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:21.163 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:21.163+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:21.437 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:21.436+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:22.016 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:22.015+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:22.140 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:22.140+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:22.476 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:22.476+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:22.992 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:22.992+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:23.101 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:23.101+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:23.485 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:23.485+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:23.991 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:23.991+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:24.139 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:24.139+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:24.523 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:24.522+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:25.019 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:25.019+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:25.112 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:25.112+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:25.501 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:25.501+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:26.046 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:26.046+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:26.081 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:26.081+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:26.516 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:26.516+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:27.074 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:27.032+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:27.074 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:27.048+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:27.556 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:27.556+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:27.996 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:27.996+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:28.053 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:28.052+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:28.517 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:28.516+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:29.021 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:29.021+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:29.042 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:29.042+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:29.544 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:29.544+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:30.017 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:30.017+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:30.058 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:30.058+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:30.550 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:30.550+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:31.011 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:31.011+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:31.057 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:31.057+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:31.510 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:31.510+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:32.034 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:32.034+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:32.105 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:32.105+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:32.555 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:32.555+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:33.058 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:33.058+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:33.064 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:33.064+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:33.532 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:33.532+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:34.090 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:34.090+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:34.111 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:34.111+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:34.569 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:34.569+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:35.093 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:35.093+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:35.100 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:35.100+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:35.548 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:35.548+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:36.080 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:36.080+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:36.109 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:36.109+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:36.526 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:36.526+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:37.081 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:37.080+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:37.118 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:37.118+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:37.566 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:37.565+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:38.094 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:38.094+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:38.142 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:38.141+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:38.596 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:38.596+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:39.075 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:39.074+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:39.176 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:39.176+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:39.556 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:39.556+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:40.111 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:40.111+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:40.193 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:40.193+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:40.566 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:40.565+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:41.156 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:41.155+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:41.192 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:41.191+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:41.584 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:41.584+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:42.132 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:42.132+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:42.205 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:42.205+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:42.624 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:42.624+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:43.121 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:43.121+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:43.186 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:43.185+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:43.616 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:43.616+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:44.092 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:44.092+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:44.203 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:44.203+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:44.576 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:44.575+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:45.138 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:45.138+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:45.201 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:45.201+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:45.528 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:45.528+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:46.166 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:46.166+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:46.187 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:46.187+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:46.571 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:46.570+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:47.134 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:47.134+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:47.141 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:47.141+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:47.601 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:47.598+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:48.117 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:48.117+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:48.125 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:48.124+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:48.633 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:48.633+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:49.149 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:49.148+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:49.175 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:49.174+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:49.612 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:49.612+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:50.144 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:50.144+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:50.208 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:50.208+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:50.578 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:50.578+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:51.112 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:51.112+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:51.160 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:51.160+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:51.570 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:51.570+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:52.111 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:52.112+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:52.133 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:52.134+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:52.534 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:52.533+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:53.136 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:53.136+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:53.176 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:53.176+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:53.526 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:53.527+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:54.091 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:54.091+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:54.150 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:54.150+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:54.571 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:54.572+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:55.051 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:55.051+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:55.133 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:55.134+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:55.577 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:55.578+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:56.026 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:56.027+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:56.151 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:56.151+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:56.530 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:56.531+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:57.071 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:57.072+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:57.156 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:57.156+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:57.512 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:57.513+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:58.033 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:58.034+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:58.188 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:58.188+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:58.491 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:58.491+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:59.064 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:08:59.065+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:59.175 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:08:59.176+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:08:59.510 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:08:59.511+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:00.017 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:00.018+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:00.140 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:00.141+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:00.480 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:00.480+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:01.036 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:01.037+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:01.107 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:01.108+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:01.482 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:01.483+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:02.007 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:02.008+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:02.156 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:02.157+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:02.524 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:02.525+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:03.038 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:03.039+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:03.203 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:03.204+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:03.564 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:03.565+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:04.084 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:04.085+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:04.166 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:04.167+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:04.562 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:04.563+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:05.055 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:05.056+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:05.156 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:05.157+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:05.527 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:05.528+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:06.028 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:06.029+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:06.199 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:06.200+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:06.540 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:06.540+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:07.000 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:07.000+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:07.174 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:07.174+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:07.571 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:07.571+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:07.998 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:07.998+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:08.211 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:08.211+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:08.563 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:08.563+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:08.964 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:08.963+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:09.256 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:09.256+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:09.602 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:09.602+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:09.941 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:09.940+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:10.232 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:10.232+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:10.593 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:10.593+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:10.896 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:10.895+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:11.266 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:11.265+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:11.547 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:11.547+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:11.898 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:11.898+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:12.251 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:12.251+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:12.520 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:12.520+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:12.875 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:12.875+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:13.208 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:13.207+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:13.558 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:13.558+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:13.858 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:13.858+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:14.207 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:14.206+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:14.566 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:14.566+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:14.846 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:14.845+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:15.160 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:15.159+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:15.583 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:15.583+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:15.809 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:15.809+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:16.124 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:16.124+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:16.601 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:16.601+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:16.846 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:16.846+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:17.128 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:17.128+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:17.594 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:17.594+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:17.859 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:17.859+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:18.126 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:18.126+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:18.639 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:18.639+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:18.898 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:18.897+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:19.131 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:19.131+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:19.672 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:19.672+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:19.934 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:19.933+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:20.131 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:20.131+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:20.638 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:20.638+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:20.976 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:20.976+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:21.099 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:21.098+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:21.682 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:21.682+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:21.998 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:21.998+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:22.139 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:22.139+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:22.699 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:22.699+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:23.026 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:23.025+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:23.133 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:23.133+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:23.720 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:23.720+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:24.064 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:24.063+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:24.174 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:24.173+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:24.677 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:24.677+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:25.107 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:25.107+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:25.140 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:25.140+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:25.715 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:25.715+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:26.148 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:26.148+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:26.170 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:26.169+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:26.682 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:26.682+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:27.182 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:27.182+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:27.217 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:27.217+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:27.660 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:27.659+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:28.202 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:28.202+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:28.239 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:28.239+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:28.670 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:28.669+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:29.178 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:29.178+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:29.257 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:29.257+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:29.645 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:29.644+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:30.164 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:30.164+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:30.234 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:30.234+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:30.605 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:30.605+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:31.131 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:31.130+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:31.275 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:31.275+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:31.591 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:31.590+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:32.116 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:32.116+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:32.260 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:32.259+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:32.595 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:32.595+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:33.125 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:33.125+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:33.301 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:33.301+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:33.550 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:33.550+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:34.149 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:34.149+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:34.279 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:34.279+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:34.567 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:34.567+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:35.181 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:35.181+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:35.273 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:35.272+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:35.520 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:35.520+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:36.200 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:36.200+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:36.277 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:36.277+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:36.493 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:36.493+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:37.239 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:37.238+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:37.256 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:37.256+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:37.458 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:37.458+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:38.194 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:38.194+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:38.233 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:38.234+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:38.427 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:38.427+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:39.188 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:39.189+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:39.223 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:39.224+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:39.457 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:39.458+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:40.206 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:40.207+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:40.215 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:40.216+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:40.451 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:40.452+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:41.208 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:41.209+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:41.230 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:41.230+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:41.496 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:41.497+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:42.204 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:42.205+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:42.228 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:42.229+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:42.514 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:42.515+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:43.200 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:43.201+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:43.204 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:43.204+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:43.538 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:43.538+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:44.206 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:44.207+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:44.238 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:44.239+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:44.548 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:44.549+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:45.220 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:45.221+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:45.227 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:45.228+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:45.582 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:45.583+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:46.216 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:46.216+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:46.235 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:46.236+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:46.596 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:46.597+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:47.198 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:47.198+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:47.202 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:47.203+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:47.572 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:47.573+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:48.186 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:48.187+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:48.216 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:48.217+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:48.529 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:48.529+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:49.153 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:49.154+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:49.177 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:49.177+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:49.500 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:49.501+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:50.139 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:50.140+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:50.170 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:50.170+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:50.503 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:50.504+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:51.092 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:51.092+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:51.162 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:51.163+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:51.526 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:51.525+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:52.063 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:52.063+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:52.202 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:52.202+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:52.523 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:52.523+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:53.024 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:53.024+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:53.235 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:53.235+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:53.493 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:53.493+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:54.035 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:54.035+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:54.196 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:54.196+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:54.470 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:54.470+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:55.057 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:55.056+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:55.230 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:55.230+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:55.423 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:55.423+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:56.037 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:56.036+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:56.248 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:56.248+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:56.418 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:56.418+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:57.008 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:57.008+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:57.206 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:57.206+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:57.457 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:57.457+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:57.970 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:57.969+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:58.245 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:58.245+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:58.426 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:58.426+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:58.930 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:58.930+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:59.244 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:09:59.244+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:59.414 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:09:59.414+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:09:59.948 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:09:59.948+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:00.268 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:00.267+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:00.399 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:00.398+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:00.949 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:00.948+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:01.228 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:01.228+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:01.357 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:01.356+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:01.986 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:01.986+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:02.259 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:02.259+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:02.308 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:02.308+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:03.002 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:03.002+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:03.246 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:03.246+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:03.332 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:03.332+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:04.016 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:04.016+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:04.230 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:04.229+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:04.370 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:04.370+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:05.039 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:05.039+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:05.232 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:05.232+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:05.362 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:05.362+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:05.991 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:05.990+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:06.224 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:06.224+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:06.336 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:06.335+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:06.954 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:06.954+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:07.226 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:07.226+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:07.300 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:07.300+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:07.923 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:07.922+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:08.270 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:08.270+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:08.333 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:08.333+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:08.935 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:08.934+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:09.239 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:09.239+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:09.288 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:09.288+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:09.929 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:09.928+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:10.248 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:10.248+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:10.330 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:10.330+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:10.963 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:10.963+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:11.286 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:11.286+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:11.375 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:11.375+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:11.969 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:11.969+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:12.296 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:12.296+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:12.343 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:12.342+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:12.941 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:12.941+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:13.299 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:13.299+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:13.340 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:13.340+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:13.954 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:13.953+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:14.327 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:14.326+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:14.332 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:14.332+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:14.938 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:14.938+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:15.315 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:15.314+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:15.328 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:15.328+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:15.973 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:15.972+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:16.356 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:16.356+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:16.365 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:16.365+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:16.939 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:16.939+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:17.385 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:17.384+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:17.394 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:17.394+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:17.963 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:17.963+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:18.386 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:18.386+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:18.429 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:18.429+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:18.934 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:18.934+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:19.408 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:19.408+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:19.445 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:19.444+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:19.912 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:19.912+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:20.444 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:20.444+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:20.488 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:20.487+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:20.864 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:20.864+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:21.450 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:21.450+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:21.485 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:21.485+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:21.829 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:21.830+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:22.444 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:22.444+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:22.477 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:22.477+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:22.877 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:22.876+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:23.400 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:23.400+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:23.492 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:23.492+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:23.914 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:23.915+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:24.419 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:24.420+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:24.541 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:24.541+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:24.884 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:24.884+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:25.442 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:25.443+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:25.577 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:25.578+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:25.907 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:25.908+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:26.403 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:26.404+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:26.558 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:26.559+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:26.954 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:26.955+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:27.361 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:27.362+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:27.550 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:27.551+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:27.958 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:27.959+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:28.398 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:28.399+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:28.561 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:28.562+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:28.979 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:28.980+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:29.376 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:29.377+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:29.519 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:29.520+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:29.988 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:29.989+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:30.422 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:30.423+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:30.522 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:30.523+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:31.031 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:31.032+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:31.448 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:31.449+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:31.476 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:31.477+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:32.039 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:32.040+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:32.466 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:32.467+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:32.489 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:32.490+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:33.073 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:33.074+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:33.441 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:33.439+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:33.484 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:33.485+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:34.079 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:34.080+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:34.422 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:34.423+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:34.508 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:34.509+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:35.036 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:35.036+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:35.447 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:35.447+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:35.546 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:35.547+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:36.075 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:36.076+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:36.428 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:36.427+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:36.528 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:36.528+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:37.063 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:37.063+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:37.412 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:37.412+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:37.569 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:37.569+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:38.087 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:38.087+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:38.379 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:38.379+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:38.539 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:38.539+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:39.071 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:39.071+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:39.360 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:39.360+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:39.540 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:39.540+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:40.110 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:40.110+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:40.385 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:40.385+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:40.590 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:40.590+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:41.065 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:41.065+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:41.416 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:41.416+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:41.544 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:41.543+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:42.081 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:42.081+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:42.381 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:42.381+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:42.510 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:42.510+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:43.065 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:43.065+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:43.430 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:43.430+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:43.556 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:43.556+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:44.105 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:44.105+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:44.464 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:44.464+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:44.550 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:44.550+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:45.132 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:45.131+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:45.484 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:45.484+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:45.564 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:45.564+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:46.167 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:46.167+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:46.520 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:46.520+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:46.540 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:46.540+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:47.153 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:47.152+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:47.478 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:47.478+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:47.497 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:47.497+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:48.167 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:48.167+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:48.441 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:48.441+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:48.453 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:48.452+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:49.175 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:49.175+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:49.428 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:49.428+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:49.462 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:49.461+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:50.152 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:50.152+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:50.416 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:50.416+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:50.484 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:50.484+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:51.199 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:51.193+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:51.444 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:51.443+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:51.487 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:51.487+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:52.230 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:52.230+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:52.398 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:52.398+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:52.486 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:52.486+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:53.251 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:53.250+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:53.410 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:53.409+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:53.458 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:53.457+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:54.239 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:54.239+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:54.398 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:54.398+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:54.412 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:54.412+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:55.237 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:55.236+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:55.367 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:55.367+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:55.430 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:55.430+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:56.249 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:56.249+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:56.393 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:56.392+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:56.453 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:56.453+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:57.223 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:57.223+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:57.378 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:57.378+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:57.416 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:57.416+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:58.227 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:58.226+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:58.369 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:58.368+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:58.400 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:58.399+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:59.205 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:10:59.202+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:59.415 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:10:59.415+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:10:59.446 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:10:59.446+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:00.244 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:00.244+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:00.375 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:00.374+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:00.476 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:00.476+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:01.271 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:01.270+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:01.404 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:01.404+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:01.497 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:01.497+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:02.293 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:02.293+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:02.364 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:02.364+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:02.492 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:02.492+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:03.248 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:03.248+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:03.335 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:03.335+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:03.522 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:03.521+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:04.277 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:04.277+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:04.320 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:04.320+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:04.550 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:04.550+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:05.229 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:05.230+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:05.274 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:05.274+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:05.585 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:05.585+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:06.238 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:06.237+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:06.271 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:06.270+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:06.631 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:06.630+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:07.243 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:07.243+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:07.266 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:07.267+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:07.589 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:07.589+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:08.208 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:08.209+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:08.249 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:08.250+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:08.544 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:08.545+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:09.225 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:09.225+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:09.226 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:09.227+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:09.526 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:09.527+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:10.195 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:10.196+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:10.226 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:10.226+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:10.576 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:10.576+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:11.244 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:11.245+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:11.264 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:11.264+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:11.533 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:11.534+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:12.223 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:12.223+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:12.226 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:12.227+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:12.494 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:12.494+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:13.219 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:13.220+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:13.275 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:13.276+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:13.512 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:13.513+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:14.200 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:14.201+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:14.274 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:14.275+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:14.489 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:14.490+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:15.172 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:15.173+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:15.239 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:15.240+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:15.540 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:15.540+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:16.190 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:16.190+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:16.213 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:16.213+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:16.540 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:16.541+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:17.178 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:17.179+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:17.197 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:17.198+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:17.556 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:17.557+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:18.150 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:18.151+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:18.190 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:18.190+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:18.514 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:18.515+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:19.159 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:19.160+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:19.222 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:19.223+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:19.554 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:19.555+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:20.182 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:20.183+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:20.208 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:20.209+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:20.528 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:20.529+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:21.221 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:21.221+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:21.251 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:21.251+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:21.569 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:21.569+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:22.217 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:22.217+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:22.225 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:22.224+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:22.593 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:22.593+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:23.195 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:23.195+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:23.246 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:23.246+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:23.555 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:23.554+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:24.237 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:24.237+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:24.246 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:24.246+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:24.510 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:24.510+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:25.211 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:25.211+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:25.262 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:25.262+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:25.502 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:25.501+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:26.229 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:26.228+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:26.235 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:26.235+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:26.468 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:26.467+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:27.236 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:27.236+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:27.245 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:27.244+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:27.490 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:27.490+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:28.238 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:28.238+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:28.261 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:28.259+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:28.458 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:28.457+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:29.223 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:29.223+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:29.267 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:29.267+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:29.421 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:29.421+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:30.212 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:30.212+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:30.263 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:30.263+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:30.394 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:30.393+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:31.199 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:31.199+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:31.259 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:31.259+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:31.443 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:31.442+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:32.170 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:32.170+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:32.221 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:32.221+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:32.398 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:32.397+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:33.182 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:33.182+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:33.198 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:33.197+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:33.388 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:33.388+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:34.204 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:34.204+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:34.243 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:34.242+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:34.384 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:34.384+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:35.210 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:35.210+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:35.233 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:35.232+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:35.356 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:35.356+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:36.166 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:36.166+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:36.278 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:36.276+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:36.320 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:36.320+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:37.156 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:37.155+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:37.281 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:37.281+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:37.337 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:37.336+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:38.168 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:38.167+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:38.288 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:38.287+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:38.376 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:38.376+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:39.214 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:39.214+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:39.280 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:39.280+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:39.420 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:39.420+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:40.216 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:40.216+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:40.294 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:40.294+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:40.373 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:40.373+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:41.211 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:41.210+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:41.316 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:41.316+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:41.329 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:41.329+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:42.238 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:42.237+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:42.290 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:42.290+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:42.359 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:42.359+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:43.200 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:43.200+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:43.259 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:43.258+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:43.364 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:43.364+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:44.163 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:44.163+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:44.227 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:44.227+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:44.361 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:44.361+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:45.181 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:45.181+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:45.251 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:45.251+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:45.346 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:45.346+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:46.216 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:46.216+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:46.235 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:46.235+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:46.356 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:46.356+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:47.211 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:47.211+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:47.264 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:47.264+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:47.376 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:47.376+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:48.231 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:48.230+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:48.235 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:48.235+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:48.370 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:48.369+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:49.227 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:49.227+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:49.255 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:49.254+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:49.416 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:49.416+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:50.209 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:50.209+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:50.284 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:50.284+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:50.388 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:50.387+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:51.181 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:51.181+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:51.327 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:51.327+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:51.400 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:51.400+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:52.208 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:52.208+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:52.363 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:52.363+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:52.388 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:52.388+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:53.169 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:53.169+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:53.392 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:53.393+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:53.395 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:53.396+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:54.140 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:54.141+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:54.352 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:54.353+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:54.422 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:54.422+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:55.186 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:55.187+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:55.326 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:55.326+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:55.460 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:55.461+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:56.147 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:56.148+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:56.322 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:56.323+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:56.469 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:56.470+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:57.115 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:57.116+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:57.304 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:57.305+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:57.482 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:57.483+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:58.066 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:58.066+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:58.314 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:58.315+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:58.471 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:58.471+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:59.060 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:11:59.061+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:59.298 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:11:59.298+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:11:59.492 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:11:59.493+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:00.070 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:00.071+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:00.283 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:00.283+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:00.520 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:00.521+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:01.082 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:01.083+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:01.306 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:01.307+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:01.566 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:01.567+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:02.074 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:02.075+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:02.346 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:02.347+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:02.576 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:02.577+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:03.066 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:03.066+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:03.298 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:03.299+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:03.622 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:03.623+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:04.109 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:04.110+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:04.293 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:04.294+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:04.650 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:04.651+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:05.093 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:05.094+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:05.313 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:05.314+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:05.680 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:05.681+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:06.140 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:06.140+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:06.350 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:06.350+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:06.707 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:06.707+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:07.153 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:07.153+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:07.334 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:07.334+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:07.728 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:07.728+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:08.181 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:08.180+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:08.352 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:08.351+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:08.734 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:08.734+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:09.142 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:09.142+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:09.382 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:09.381+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:09.690 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:09.690+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:10.134 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:10.134+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:10.354 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:10.354+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:10.664 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:10.664+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:11.156 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:11.156+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:11.353 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:11.353+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:11.675 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:11.674+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:12.141 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:12.140+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:12.323 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:12.323+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:12.723 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:12.722+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:13.097 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:13.097+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:13.326 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:13.326+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:13.728 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:13.728+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:14.071 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:14.071+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:14.296 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:14.296+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:14.710 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:14.710+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:15.081 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:15.081+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:15.308 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:15.308+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:15.737 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:15.737+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:16.054 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:16.054+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:16.290 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:16.290+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:16.757 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:16.757+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:17.025 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:17.025+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:17.286 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:17.286+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:17.736 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:17.736+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:18.007 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:18.006+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:18.264 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:18.263+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:18.706 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:18.705+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:19.049 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:19.049+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:19.242 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:19.242+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:19.706 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:19.706+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:20.045 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:20.045+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:20.219 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:20.219+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:20.658 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:20.658+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:21.013 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:21.013+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:21.202 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:21.202+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:21.616 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:21.615+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:21.985 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:21.984+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:22.159 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:22.159+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:22.656 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:22.656+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:23.025 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:23.025+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:23.163 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:23.163+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:23.653 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:23.652+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:23.986 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:23.985+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:24.194 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:24.194+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:24.634 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:24.634+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:24.937 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:24.937+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:25.220 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:25.220+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:25.648 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:25.648+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:25.896 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:25.896+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:26.242 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:26.241+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:26.603 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:26.603+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:26.867 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:26.867+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:27.212 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:27.212+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:27.610 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:27.610+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:27.911 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:27.911+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:28.223 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:28.223+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:28.591 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:28.591+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:28.922 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:28.921+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:29.255 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:29.255+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:29.600 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:29.599+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:29.903 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:29.903+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:30.268 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:30.268+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:30.579 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:30.579+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:30.932 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:30.932+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:31.255 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:31.255+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:31.583 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:31.582+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:31.943 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:31.943+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:32.294 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:32.294+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:32.616 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:32.616+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:32.992 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:32.991+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:33.308 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:33.308+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:33.635 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:33.635+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:34.039 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:34.039+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:34.352 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:34.352+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:34.634 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:34.634+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:35.029 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:35.030+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:35.358 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:35.357+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:35.627 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:35.627+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:36.039 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:36.039+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:36.359 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:36.358+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:36.636 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:36.637+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:37.004 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:37.005+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:37.313 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:37.313+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:37.663 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:37.664+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:38.025 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:38.026+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:38.350 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:38.351+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:38.658 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:38.659+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:39.007 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:39.008+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:39.389 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:39.390+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:39.622 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:39.623+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:40.054 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:40.055+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:40.438 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:40.439+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:40.636 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:40.637+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:41.014 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:41.015+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:41.416 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:41.417+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:41.604 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:41.605+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:42.022 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:42.023+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:42.456 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:42.457+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:42.625 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:42.626+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:42.996 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:42.997+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:43.450 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:43.451+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:43.603 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:43.604+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:44.038 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:44.039+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:44.414 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:44.415+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:44.590 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:44.591+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:45.064 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:45.065+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:45.388 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:45.389+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:45.578 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:45.579+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:46.027 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:46.027+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:46.412 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:46.413+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:46.531 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:46.532+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:47.073 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:47.073+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:47.440 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:47.440+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:47.487 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:47.488+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:48.107 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:48.107+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:48.468 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:48.469+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:48.468 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:48.469+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:49.091 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:49.092+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:49.464 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:49.465+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:49.472 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:49.472+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:50.055 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:50.055+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:50.436 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:50.436+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:50.478 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:50.479+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:51.065 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:51.065+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:51.400 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:51.400+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:51.511 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:51.511+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:52.105 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:52.104+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:52.438 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:52.438+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:52.478 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:52.477+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:53.099 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:53.099+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:53.392 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:53.392+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:53.480 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:53.479+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:54.067 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:54.067+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:54.370 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:54.370+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:54.508 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:54.507+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:55.053 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:55.052+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:55.369 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:55.368+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:55.463 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:55.463+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:56.075 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:56.075+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:56.399 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:56.398+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:56.416 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:56.415+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:57.095 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:57.095+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:57.376 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:57.376+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:57.440 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:57.440+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:58.062 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:58.062+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:58.389 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:58.389+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:58.464 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:58.464+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:59.095 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:12:59.094+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:59.356 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:12:59.355+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:12:59.482 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:12:59.480+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:00.090 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:00.090+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:00.323 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:00.323+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:00.490 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:00.490+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:01.064 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:01.064+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:01.292 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:01.291+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:01.498 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:01.498+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:02.046 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:02.045+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:02.323 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:02.323+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:02.535 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:02.535+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:03.088 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:03.087+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:03.350 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:03.350+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:03.517 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:03.517+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:04.055 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:04.054+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:04.316 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:04.315+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:04.499 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:04.499+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:05.018 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:05.018+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:05.290 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:05.290+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:05.470 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:05.470+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:06.002 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:06.001+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:06.315 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:06.314+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:06.485 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:06.485+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:07.038 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:07.037+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:07.346 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:07.346+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:07.445 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:07.445+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:08.031 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:08.031+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:08.362 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:08.362+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:08.476 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:08.476+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:08.994 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:08.994+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:09.363 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:09.363+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:09.456 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:09.456+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:10.033 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:10.033+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:10.316 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:10.316+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:10.475 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:10.474+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:11.059 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:11.059+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:11.348 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:11.347+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:11.463 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:11.463+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:12.033 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:12.033+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:12.313 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:12.313+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:12.418 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:12.418+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:13.022 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:13.022+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:13.271 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:13.271+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:13.424 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:13.424+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:13.996 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:13.996+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:14.301 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:14.301+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:14.396 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:14.396+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:14.954 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:14.954+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:15.262 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:15.262+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:15.408 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:15.408+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:15.961 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:15.960+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:16.252 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:16.252+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:16.420 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:16.420+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:16.959 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:16.959+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:17.284 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:17.284+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:17.407 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:17.407+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:17.939 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:17.939+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:18.299 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:18.299+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:18.390 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:18.390+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:18.986 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:18.985+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:19.257 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:19.257+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:19.431 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:19.430+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:20.003 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:20.003+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:20.211 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:20.211+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:20.436 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:20.436+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:21.021 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:21.021+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:21.248 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:21.247+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:21.473 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:21.473+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:21.973 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:21.973+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:22.297 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:22.296+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:22.520 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:22.520+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:22.954 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:22.955+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:23.271 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:23.271+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:23.476 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:23.477+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:23.973 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:23.974+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:24.292 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:24.293+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:24.514 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:24.514+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:24.978 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:24.979+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:25.262 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:25.262+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:25.516 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:25.517+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:26.003 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:26.003+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:26.248 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:26.249+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:26.526 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:26.527+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:27.011 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:27.011+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:27.207 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:27.208+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:27.505 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:27.505+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:28.041 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:28.042+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:28.162 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:28.163+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:28.462 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:28.463+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:29.039 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:29.039+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:29.192 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:29.192+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:29.449 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:29.449+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:29.993 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:29.994+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:30.200 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:30.201+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:30.478 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:30.479+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:31.011 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:31.012+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:31.171 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:31.172+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:31.488 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:31.489+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:31.967 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:31.967+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:32.190 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:32.191+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:32.515 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:32.516+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:32.999 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:33.000+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:33.180 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:33.181+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:33.479 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:33.480+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:33.961 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:33.962+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:34.157 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:34.157+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:34.498 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:34.499+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:34.978 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:34.979+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:35.188 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:35.189+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:35.548 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:35.548+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:35.992 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:35.992+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:36.176 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:36.176+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:36.514 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:36.514+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:36.957 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:36.957+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:37.212 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:37.211+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:37.487 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:37.487+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:37.950 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:37.950+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:38.198 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:38.198+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:38.535 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:38.534+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:38.926 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:38.926+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:39.173 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:39.172+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:39.500 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:39.500+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:39.970 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:39.970+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:40.217 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:40.216+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:40.469 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:40.468+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:40.955 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:40.955+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:41.263 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:41.263+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:41.455 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:41.455+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:41.930 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:41.930+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:42.288 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:42.287+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:42.483 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:42.483+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:42.956 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:42.955+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:43.294 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:43.294+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:43.529 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:43.529+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:43.938 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:43.938+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:44.289 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:44.288+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:44.568 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:44.568+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:44.899 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:44.899+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:45.320 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:45.319+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:45.613 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:45.612+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:45.888 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:45.888+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:46.308 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:46.307+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:46.607 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:46.607+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:46.848 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:46.848+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:47.348 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:47.348+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:47.587 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:47.586+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:47.867 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:47.867+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:48.324 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:48.324+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:48.576 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:48.576+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:48.879 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:48.879+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:49.298 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:49.298+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:49.582 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:49.581+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:49.909 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:49.908+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:50.330 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:50.330+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:50.546 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:50.546+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:50.923 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:50.923+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:51.359 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:51.358+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:51.580 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:51.579+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:51.898 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:51.898+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:52.398 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:52.398+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:52.601 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:52.601+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:52.860 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:52.860+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:53.436 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:53.435+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:53.651 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:53.650+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:53.905 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:53.905+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:54.451 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:54.451+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:54.621 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:54.620+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:54.907 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:54.907+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:55.472 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:55.472+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:55.632 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:55.632+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:55.916 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:55.916+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:56.469 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:56.469+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:56.587 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:56.587+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:56.949 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:56.948+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:57.436 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:57.436+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:57.548 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:57.548+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:57.982 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:57.982+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:58.455 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:58.455+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:58.557 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:58.557+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:59.015 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:13:59.015+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:59.437 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:13:59.436+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:13:59.536 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:13:59.536+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:00.026 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:00.026+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:00.408 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:00.407+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:00.568 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:00.568+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:01.008 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:01.007+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:01.423 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:01.422+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:01.593 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:01.589+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:02.055 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:02.055+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:02.436 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:02.435+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:02.630 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:02.630+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:03.055 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:03.055+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:03.420 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:03.420+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:03.613 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:03.613+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:04.021 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:04.021+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:04.469 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:04.469+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:04.625 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:04.625+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:05.031 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:05.030+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:05.446 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:05.445+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:05.584 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:05.583+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:06.034 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:06.034+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:06.446 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:06.446+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:06.631 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:06.631+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:07.075 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:07.074+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:07.446 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:07.446+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:07.598 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:07.598+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:08.074 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:08.074+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:08.495 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:08.496+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:08.553 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:08.554+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:09.044 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:09.045+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:09.469 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:09.470+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:09.601 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:09.602+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:10.020 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:10.021+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:10.445 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:10.446+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:10.605 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:10.606+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:11.006 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:11.007+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:11.425 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:11.426+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:11.598 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:11.599+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:12.040 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:12.041+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:12.394 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:12.395+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:12.610 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:12.611+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:13.055 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:13.056+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:13.439 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:13.440+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:13.573 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:13.574+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:14.024 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:14.025+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:14.407 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:14.408+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:14.575 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:14.576+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:15.023 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:15.024+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:15.372 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:15.373+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:15.560 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:15.560+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:16.044 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:16.045+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:16.332 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:16.332+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:16.576 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:16.577+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:17.008 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:17.009+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:17.292 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:17.293+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:17.595 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:17.596+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:18.029 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:18.029+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:18.294 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:18.294+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:18.585 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:18.586+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:19.066 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:19.067+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:19.336 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:19.337+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:19.583 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:19.584+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:20.055 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:20.056+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:20.363 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:20.364+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:20.627 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:20.627+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:21.055 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:21.054+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:21.345 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:21.345+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:21.618 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:21.618+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:22.028 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:22.028+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:22.349 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:22.348+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:22.609 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:22.608+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:22.987 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:22.987+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:23.371 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:23.371+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:23.651 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:23.651+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:23.975 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:23.975+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:24.376 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:24.375+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:24.654 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:24.654+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:24.953 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:24.952+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:25.392 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:25.391+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:25.699 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:25.698+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:25.985 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:25.985+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:26.417 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:26.416+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:26.675 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:26.675+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:26.960 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:26.959+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:27.417 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:27.416+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:27.652 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:27.651+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:27.966 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:27.966+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:28.425 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:28.425+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:28.618 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:28.617+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:28.972 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:28.971+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:29.444 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:29.444+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:29.641 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:29.640+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:29.939 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:29.939+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:30.484 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:30.481+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:30.653 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:30.653+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:30.918 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:30.917+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:31.448 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:31.448+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:31.629 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:31.629+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:31.941 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:31.941+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:32.467 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:32.467+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:32.637 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:32.637+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:32.967 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:32.967+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:33.449 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:33.449+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:33.603 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:33.603+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:33.989 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:33.988+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:34.428 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:34.427+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:34.555 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:34.555+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:35.007 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:35.007+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:35.478 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:35.478+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:35.604 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:35.604+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:36.001 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:36.001+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:36.478 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:36.478+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:36.642 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:36.642+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:36.998 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:36.998+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:37.486 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:37.486+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:37.608 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:37.607+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:38.000 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:38.000+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:38.449 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:38.448+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:38.607 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:38.606+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:39.002 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:39.002+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:39.452 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:39.451+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:39.567 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:39.567+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:40.038 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:40.038+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:40.488 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:40.488+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:40.575 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:40.574+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:40.995 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:40.995+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:41.450 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:41.450+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:41.612 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:41.612+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:41.960 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:41.959+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:42.420 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:42.420+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:42.610 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:42.609+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:42.941 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:42.940+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:43.451 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:43.451+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:43.564 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:43.563+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:43.939 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:43.939+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:44.480 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:44.479+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:44.582 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:44.582+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:44.935 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:44.934+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:45.512 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:45.512+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:45.607 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:45.607+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:45.956 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:45.956+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:46.545 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:46.545+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:46.623 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:46.623+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:46.920 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:46.920+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:47.531 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:47.530+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:47.669 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:47.669+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:47.944 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:47.944+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:48.551 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:48.551+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:48.719 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:48.719+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:48.906 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:48.906+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:49.578 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:49.578+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:49.700 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:49.700+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:49.857 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:49.857+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:50.539 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:50.539+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:50.660 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:50.660+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:50.852 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:50.851+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:51.530 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:51.530+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:51.645 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:51.645+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:51.831 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:51.831+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:52.560 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:52.561+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:52.694 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:52.695+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:52.823 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:52.823+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:53.533 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:53.534+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:53.669 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:53.670+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:53.797 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:53.798+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:54.575 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:54.575+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:54.709 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:54.710+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:54.764 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:54.765+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:55.617 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:55.618+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:55.691 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:55.692+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:55.741 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:55.741+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:56.633 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:56.633+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:56.659 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:56.660+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:56.755 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:56.756+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:57.605 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:57.606+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:57.676 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:57.677+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:57.737 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:57.738+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:58.601 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:58.601+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:58.674 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:58.675+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:58.717 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:58.717+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:59.606 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:14:59.606+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:59.638 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:14:59.639+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:14:59.680 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:14:59.681+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:00.617 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:00.618+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:00.633 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:00.634+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:00.644 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:00.644+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:01.614 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:01.614+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:01.648 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:01.648+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:01.660 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:01.661+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:02.601 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:02.601+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:02.632 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:02.633+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:02.643 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:02.644+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:03.601 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:03.602+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:03.647 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:03.648+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:03.670 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:03.671+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:04.599 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:04.600+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:04.631 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:04.632+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:04.711 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:04.711+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:05.645 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:05.644+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:05.662 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:05.661+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:05.684 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:05.684+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:06.602 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:06.602+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:06.653 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:06.653+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:06.698 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:06.698+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:07.613 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:07.613+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:07.660 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:07.660+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:07.698 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:07.697+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:08.592 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:08.592+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:08.666 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:08.666+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:08.693 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:08.693+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:09.630 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:09.629+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:09.686 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:09.685+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:09.716 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:09.716+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:10.605 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:10.605+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:10.666 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:10.666+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:10.748 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:10.748+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:11.555 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:11.555+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:11.618 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:11.618+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:11.798 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:11.798+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:12.559 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:12.558+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:12.616 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:12.616+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:12.823 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:12.823+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:13.527 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:13.527+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:13.587 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:13.587+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:13.775 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:13.775+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:14.548 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:14.548+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:14.566 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:14.564+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:14.770 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:14.769+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:15.516 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:15.516+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:15.575 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:15.575+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:15.794 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:15.793+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:16.496 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:16.496+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:16.593 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:16.593+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:16.840 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:16.840+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:17.490 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:17.490+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:17.546 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:17.546+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:17.880 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:17.880+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:18.443 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:18.443+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:18.592 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:18.592+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:18.914 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:18.914+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:19.441 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:19.441+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:19.630 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:19.630+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:19.874 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:19.873+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:20.439 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:20.439+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:20.614 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:20.614+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:20.892 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:20.892+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:21.411 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:21.410+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:21.589 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:21.588+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:21.885 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:21.884+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:22.431 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:22.431+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:22.580 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:22.580+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:22.929 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:22.928+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:23.439 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:23.438+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:23.620 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:23.620+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:23.914 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:23.914+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:24.481 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:24.480+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:24.609 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:24.609+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:24.869 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:24.869+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:25.474 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:25.473+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:25.596 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:25.595+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:25.909 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:25.909+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:26.506 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:26.505+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:26.586 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:26.585+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:26.917 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:26.917+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:27.551 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:27.550+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:27.591 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:27.591+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:27.872 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:27.872+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:28.519 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:28.519+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:28.560 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:28.559+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:28.906 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:28.906+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:29.471 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:29.470+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:29.560 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:29.559+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:29.908 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:29.907+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:30.447 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:30.447+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:30.518 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:30.517+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:30.944 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:30.944+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:31.479 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:31.479+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:31.514 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:31.514+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:31.936 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:31.936+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:32.525 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:32.524+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:32.530 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:32.530+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:32.953 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:32.953+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:33.527 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:33.526+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:33.531 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:33.531+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:33.915 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:33.915+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:34.565 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:34.565+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:34.572 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:34.572+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:34.944 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:34.944+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:35.591 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:35.591+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:35.599 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:35.599+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:35.943 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:35.942+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:36.577 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:36.577+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:36.639 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:36.639+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:36.931 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:36.931+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:37.604 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:37.604+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:37.666 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:37.667+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:37.962 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:37.963+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:38.624 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:38.625+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:38.670 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:38.671+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:38.946 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:38.947+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:39.653 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:39.654+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:39.672 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:39.672+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:39.962 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:39.963+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:40.697 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:40.698+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:40.714 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:40.714+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:40.993 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:40.993+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:41.674 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:41.675+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:41.722 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:41.723+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:41.978 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:41.979+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:42.673 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:42.673+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:42.733 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:42.734+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:42.965 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:42.966+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:43.651 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:43.652+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:43.746 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:43.747+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:43.996 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:43.997+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:44.617 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:44.618+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:44.698 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:44.698+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:44.947 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:44.947+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:45.616 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:45.616+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:45.688 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:45.689+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:45.987 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:45.988+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:46.590 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:46.591+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:46.662 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:46.662+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:47.036 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:47.036+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:47.549 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:47.549+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:47.671 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:47.672+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:48.076 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:48.077+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:48.506 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:48.506+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:48.669 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:48.670+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:49.034 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:49.035+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:49.468 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:49.469+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:49.627 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:49.628+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:50.063 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:50.063+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:50.466 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:50.466+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:50.623 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:50.623+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:51.034 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:51.034+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:51.431 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:51.431+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:51.605 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:51.604+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:52.010 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:52.010+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:52.402 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:52.402+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:52.638 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:52.637+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:53.059 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:53.059+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:53.448 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:53.447+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:53.669 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:53.668+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:54.024 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:54.024+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:54.462 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:54.461+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:54.638 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:54.638+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:55.041 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:55.040+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:55.418 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:55.418+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:55.611 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:55.611+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:56.085 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:56.084+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:56.448 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:56.448+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:56.619 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:56.619+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:57.049 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:57.049+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:57.452 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:57.452+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:57.648 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:57.648+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:58.047 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:58.046+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:58.420 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:58.420+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:58.619 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:58.619+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:59.080 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:15:59.080+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:59.467 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:15:59.467+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:15:59.663 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:15:59.663+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:00.054 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:00.054+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:00.426 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:00.426+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:00.708 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:00.707+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:01.058 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:01.058+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:01.432 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:01.432+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:01.690 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:01.689+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:02.109 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:02.108+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:02.436 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:02.435+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:02.679 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:02.679+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:03.104 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:03.104+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:03.426 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:03.426+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:03.673 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:03.672+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:04.074 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:04.074+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:04.424 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:04.424+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:04.693 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:04.693+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:05.119 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:05.118+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:05.472 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:05.472+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:05.645 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:05.644+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:06.139 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:06.139+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:06.478 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:06.478+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:06.652 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:06.652+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:07.116 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:07.115+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:07.454 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:07.454+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:07.619 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:07.619+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:08.097 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:08.097+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:08.462 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:08.461+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:08.579 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:08.578+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:09.145 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:09.145+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:09.444 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:09.444+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:09.621 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:09.621+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:10.138 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:10.138+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:10.487 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:10.487+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:10.639 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:10.639+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:11.100 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:11.099+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:11.520 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:11.520+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:11.611 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:11.611+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:12.085 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:12.084+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:12.478 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:12.478+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:12.629 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:12.629+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:13.058 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:13.058+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:13.478 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:13.478+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:13.639 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:13.639+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:14.023 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:14.023+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:14.470 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:14.470+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:14.593 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:14.592+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:15.052 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:15.051+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:15.492 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:15.492+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:15.594 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:15.594+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:16.057 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:16.057+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:16.510 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:16.509+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:16.597 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:16.597+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:17.036 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:17.036+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:17.560 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:17.559+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:17.573 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:17.573+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:18.086 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:18.086+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:18.543 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:18.543+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:18.583 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:18.583+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:19.039 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:19.038+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:19.553 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:19.553+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:19.601 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:19.601+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:19.995 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:19.995+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:20.550 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:20.549+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:20.645 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:20.645+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:21.034 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:21.033+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:21.514 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:21.514+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:21.598 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:21.598+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:22.010 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:22.010+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:22.473 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:22.474+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:22.596 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:22.597+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:22.994 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:22.994+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:23.429 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:23.430+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:23.597 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:23.598+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:24.022 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:24.022+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:24.459 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:24.460+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:24.639 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:24.639+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:25.003 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:25.004+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:25.442 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:25.443+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:25.667 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:25.668+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:25.981 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:25.981+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:26.460 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:26.461+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:26.707 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:26.708+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:27.005 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:27.005+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:27.439 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:27.439+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:27.669 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:27.669+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:27.996 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:27.996+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:28.477 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:28.477+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:28.709 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:28.709+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:29.037 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:29.037+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:29.468 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:29.469+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:29.706 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:29.706+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:30.055 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:30.055+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:30.512 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:30.513+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:30.697 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:30.697+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:31.014 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:31.015+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:31.488 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:31.489+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:31.702 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:31.703+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:32.031 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:32.032+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:32.530 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:32.531+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:32.663 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:32.664+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:32.999 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:32.999+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:33.519 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:33.519+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:33.709 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:33.710+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:34.027 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:34.027+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:34.491 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:34.492+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:34.680 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:34.680+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:34.985 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:34.986+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:35.513 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:35.513+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:35.716 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:35.716+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:36.033 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:36.033+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:36.508 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:36.508+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:36.745 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:36.745+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:37.022 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:37.021+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:37.536 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:37.536+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:37.749 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:37.749+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:37.995 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:37.995+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:38.517 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:38.517+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:38.739 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:38.739+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:38.992 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:38.991+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:39.470 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:39.470+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:39.746 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:39.745+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:39.975 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:39.975+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:40.512 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:40.512+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:40.781 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:40.778+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:40.980 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:40.979+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:41.536 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:41.535+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:41.789 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:41.788+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:41.979 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:41.979+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:42.512 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:42.512+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:42.836 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:42.836+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:43.006 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:43.006+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:43.524 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:43.524+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:43.811 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:43.810+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:44.013 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:44.012+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:44.543 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:44.543+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:44.784 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:44.783+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:45.046 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:45.046+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:45.503 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:45.503+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:45.745 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:45.745+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:46.053 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:46.053+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:46.454 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:46.453+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:46.795 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:46.795+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:47.037 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:47.037+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:47.502 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:47.502+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:47.825 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:47.825+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:48.082 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:48.082+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:48.542 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:48.542+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:48.858 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:48.857+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:49.124 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:49.124+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:49.502 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:49.502+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:49.851 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:49.850+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:50.099 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:50.099+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:50.491 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:50.491+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:50.851 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:50.851+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:51.149 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:51.149+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:51.534 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:51.533+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:51.898 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:51.898+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:52.179 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:52.179+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:52.521 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:52.521+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:52.946 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:52.946+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:53.172 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:53.172+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:53.476 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:53.476+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:53.971 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:53.970+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:54.177 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:54.177+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:54.436 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:54.436+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:54.974 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:54.973+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:55.189 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:55.189+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:55.451 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:55.451+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:55.985 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:55.985+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:56.176 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:56.174+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:56.430 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:56.430+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:56.997 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:56.997+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:57.217 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:57.216+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:57.399 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:57.398+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:57.994 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:57.994+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:58.197 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:58.196+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:58.449 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:58.448+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:59.013 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:16:59.013+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:59.207 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:16:59.207+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:16:59.477 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:16:59.477+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:00.045 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:00.044+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:00.251 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:00.251+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:00.513 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:00.512+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:01.028 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:01.027+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:01.207 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:01.207+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:01.466 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:01.465+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:01.995 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:01.995+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:02.161 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:02.160+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:02.467 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:02.467+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:03.046 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:03.046+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:03.177 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:03.177+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:03.440 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:03.440+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:04.043 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:04.042+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:04.207 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:04.207+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:04.447 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:04.448+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:05.092 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:05.092+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:05.197 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:05.197+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:05.407 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:05.406+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:06.139 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:06.139+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:06.194 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:06.194+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:06.442 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:06.443+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:07.102 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:07.103+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:07.214 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:07.215+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:07.407 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:07.408+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:08.078 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:08.079+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:08.180 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:08.181+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:08.382 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:08.383+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:09.104 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:09.104+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:09.148 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:09.148+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:09.360 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:09.361+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:10.082 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:10.083+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:10.154 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:10.155+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:10.410 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:10.411+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:11.108 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:11.108+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:11.194 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:11.194+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:11.387 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:11.388+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:12.084 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:12.085+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:12.201 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:12.202+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:12.382 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:12.383+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:13.047 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:13.048+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:13.160 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:13.161+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:13.392 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:13.393+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:14.043 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:14.044+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:14.144 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:14.145+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:14.404 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:14.405+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:15.028 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:15.029+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:15.169 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:15.169+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:15.407 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:15.408+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:16.069 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:16.069+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:16.195 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:16.196+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:16.406 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:16.407+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:17.059 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:17.060+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:17.187 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:17.188+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:17.370 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:17.371+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:18.025 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:18.026+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:18.180 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:18.181+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:18.345 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:18.346+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:19.006 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:19.007+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:19.171 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:19.171+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:19.340 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:19.340+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:20.018 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:20.017+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:20.145 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:20.145+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:20.355 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:20.355+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:20.974 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:20.974+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:21.138 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:21.138+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:21.350 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:21.350+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:21.981 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:21.980+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:22.129 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:22.129+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:22.321 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:22.321+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:23.018 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:23.017+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:23.158 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:23.158+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:23.345 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:23.344+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:23.988 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:23.988+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:24.201 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:24.200+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:24.345 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:24.344+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:25.010 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:25.010+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:25.158 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:25.158+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:25.336 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:25.336+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:25.983 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:25.982+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:26.139 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:26.138+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:26.289 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:26.289+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:26.976 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:26.975+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:27.091 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:27.091+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:27.264 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:27.264+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:27.991 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:27.991+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:28.084 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:28.084+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:28.282 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:28.281+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:28.944 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:28.943+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:29.131 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:29.130+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:29.239 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:29.239+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:29.956 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:29.956+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:30.171 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:30.171+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:30.235 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:30.235+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:30.973 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:30.973+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:31.218 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:31.218+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:31.233 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:31.233+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:31.942 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:31.941+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:32.230 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:32.229+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:32.266 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:32.266+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:32.928 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:32.928+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:33.236 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:33.235+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:33.294 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:33.294+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:33.935 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:33.934+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:34.221 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:34.221+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:34.290 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:34.290+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:34.892 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:34.892+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:35.222 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:35.221+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:35.298 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:35.297+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:35.929 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:35.928+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:36.223 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:36.222+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:36.287 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:36.287+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:36.922 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:36.922+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:37.193 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:37.192+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:37.264 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:37.264+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:37.926 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:37.926+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:38.148 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:38.148+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:38.224 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:38.224+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:38.942 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:38.942+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:39.099 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:39.099+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:39.254 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:39.254+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:39.911 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:39.910+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:40.079 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:40.079+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:40.279 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:40.279+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:40.885 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:40.885+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:41.114 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:41.114+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 8 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:41.295 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:41.295+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:41.930 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:41.930+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:42.066 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:42.066+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 7 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:42.252 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:42.252+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:42.892 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:42.892+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:43.028 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:43.027+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 7 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:43.264 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:43.264+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:43.937 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:43.937+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:43.985 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:43.985+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 7 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:44.276 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:44.276+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:44.920 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:44.920+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:44.963 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:44.963+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 7 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:45.283 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:45.283+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:45.907 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:45.907+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:45.915 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:45.915+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 7 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:46.328 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:46.327+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:46.881 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:46.881+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:46.882 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:46.882+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 7 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:47.368 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:47.368+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:47.840 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:47.840+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 7 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:47.841 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:47.841+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:48.357 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:48.357+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:48.876 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:48.875+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:48.877 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:48.877+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 7 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:49.359 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:49.359+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:49.843 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:49.843+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 7 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:49.856 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:49.855+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:50.367 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:50.366+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:50.822 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:50.822+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 7 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:50.829 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:50.829+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:51.412 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:51.412+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:51.784 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:51.785+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 7 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:51.799 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:51.800+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:52.385 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:52.385+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:52.746 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:52.746+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 7 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:52.807 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:52.808+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:53.397 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:53.398+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:53.775 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:53.776+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 7 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:53.807 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:53.807+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:54.428 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:54.429+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:54.752 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:54.753+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 7 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:54.780 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:54.781+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:55.470 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:55.471+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:55.759 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:55.759+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:55.769 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:55.770+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 7 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:56.504 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:56.504+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:56.735 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:56.735+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 7 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:56.756 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:56.757+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:57.492 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:57.493+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:57.693 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:57.693+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 7 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:57.734 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:57.734+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:58.468 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:58.469+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:58.716 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:58.717+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 7 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:58.728 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:58.729+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:59.507 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:17:59.508+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:59.716 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:17:59.717+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:17:59.745 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:17:59.746+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 7 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:00.471 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:00.472+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:00.690 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:00.690+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:00.766 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:00.767+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 7 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:01.510 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:01.511+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:01.693 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:01.694+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:01.754 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:01.755+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 7 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:02.476 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:02.477+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:02.714 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:02.714+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 7 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:02.721 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:02.722+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:03.514 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:03.514+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:03.688 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:03.689+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 7 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:03.719 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:03.719+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:04.560 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:04.561+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:04.686 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:04.686+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:04.691 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:04.691+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 7 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:05.580 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:05.580+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:05.650 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:05.650+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 7 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:05.731 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:05.730+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:06.542 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:06.542+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:06.691 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:06.690+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 7 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:06.738 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:06.738+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:07.503 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:07.503+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:07.688 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:07.688+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 7 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:07.788 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:07.788+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:08.465 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:08.465+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:08.730 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:08.730+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 7 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:08.774 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:08.772+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:09.441 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:09.440+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:09.704 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:09.704+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 7 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:09.774 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:09.774+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:10.452 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:10.452+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:10.656 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:10.656+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 7 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:10.763 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:10.763+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:11.455 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:11.455+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:11.606 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:11.606+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:11.775 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:11.774+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:12.493 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:12.493+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:12.583 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:12.582+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:12.815 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:12.815+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:13.532 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:13.531+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:13.568 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:13.568+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:13.802 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:13.798+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:14.526 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:14.526+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:14.600 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:14.599+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:14.812 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:14.812+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:15.499 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:15.499+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:15.619 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:15.619+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:15.826 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:15.825+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:16.475 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:16.475+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:16.618 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:16.618+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:16.816 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:16.816+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:17.478 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:17.478+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:17.661 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:17.661+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:17.802 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:17.802+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:18.493 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:18.493+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:18.668 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:18.668+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:18.842 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:18.842+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:19.536 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:19.535+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:19.669 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:19.668+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:19.794 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:19.794+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:20.492 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:20.491+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:20.694 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:20.694+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:20.788 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:20.788+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:21.490 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:21.490+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:21.677 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:21.677+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:21.798 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:21.798+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:22.462 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:22.462+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:22.674 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:22.673+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:22.772 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:22.772+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:23.473 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:23.472+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:23.668 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:23.668+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:23.802 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:23.802+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:24.476 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:24.476+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:24.673 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:24.672+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:24.847 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:24.846+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:25.433 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:25.432+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:25.687 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:25.687+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:25.871 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:25.871+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:26.439 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:26.439+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:26.660 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:26.660+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:26.826 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:26.826+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:27.410 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:27.409+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:27.699 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:27.699+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:27.792 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:27.792+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:28.416 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:28.416+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:28.717 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:28.716+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:28.747 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:28.747+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:29.412 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:29.412+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:29.701 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:29.700+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:29.712 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:29.712+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:30.459 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:30.458+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:30.657 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:30.656+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:30.730 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:30.730+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:31.446 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:31.446+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:31.698 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:31.698+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:31.741 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:31.741+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:32.488 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:32.488+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:32.682 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:32.682+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:32.726 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:32.725+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:33.529 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:33.529+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:33.639 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:33.639+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:33.682 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:33.682+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:34.540 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:34.540+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:34.621 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:34.621+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:34.667 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:34.666+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:35.495 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:35.494+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:35.649 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:35.650+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:35.714 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:35.714+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:36.513 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:36.513+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:36.645 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:36.646+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:36.737 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:36.738+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:37.550 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:37.551+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:37.649 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:37.649+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:37.771 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:37.772+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:38.582 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:38.583+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:38.676 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:38.677+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:38.810 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:38.811+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:39.602 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:39.603+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:39.642 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:39.643+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:39.861 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:39.861+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:40.563 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:40.564+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:40.613 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:40.613+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:40.899 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:40.900+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:41.568 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:41.568+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:41.594 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:41.595+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:41.899 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:41.900+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:42.549 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:42.550+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:42.598 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:42.599+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:42.944 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:42.945+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:43.506 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:43.507+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:43.639 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:43.639+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:43.931 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:43.932+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:44.497 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:44.498+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:44.649 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:44.649+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:44.979 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:44.980+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:45.495 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:45.496+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:45.639 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:45.640+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:45.950 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:45.951+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:46.463 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:46.464+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:46.688 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:46.688+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:46.970 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:46.970+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:47.432 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:47.433+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:47.673 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:47.673+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:47.997 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:47.998+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:48.385 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:48.386+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:48.705 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:48.705+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:48.978 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:48.978+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:49.405 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:49.405+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:49.665 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:49.665+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:50.022 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:50.022+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:50.445 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:50.444+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:50.639 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:50.639+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:50.997 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:50.996+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:51.405 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:51.405+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:51.650 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:51.649+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:51.962 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:51.961+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:52.454 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:52.453+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:52.658 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:52.658+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:52.921 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:52.921+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:53.447 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:53.447+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:53.628 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:53.628+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:53.893 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:53.893+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:54.420 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:54.420+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:54.631 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:54.631+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:54.883 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:54.883+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:55.394 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:55.394+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:55.628 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:55.627+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:55.873 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:55.873+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:56.400 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:56.400+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:56.589 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:56.589+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:56.841 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:56.841+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:57.423 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:57.423+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:57.588 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:57.588+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:57.876 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:57.875+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:58.445 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:58.445+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:58.581 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:58.581+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:58.856 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:58.856+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:59.494 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:18:59.494+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:59.561 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:18:59.560+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:18:59.878 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:18:59.878+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:00.490 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:00.489+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:00.584 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:00.583+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:00.879 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:00.879+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:01.498 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:01.498+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:01.545 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:01.545+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:01.919 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:01.919+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:02.503 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:02.503+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:02.526 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:02.526+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:02.899 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:02.898+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:03.549 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:03.549+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:03.559 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:03.559+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:03.936 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:03.936+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:04.553 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:04.553+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:04.569 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:04.569+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:04.968 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:04.967+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:05.560 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:05.559+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:05.615 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:05.614+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:06.006 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:06.005+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:06.569 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:06.569+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:06.624 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:06.624+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:06.981 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:06.980+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:07.558 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:07.558+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:07.588 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:07.588+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:07.947 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:07.947+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:08.559 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:08.558+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:08.572 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:08.572+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:08.905 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:08.904+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:09.574 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:09.574+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:09.590 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:09.590+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:09.921 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:09.920+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:10.572 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:10.572+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:10.582 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:10.582+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:10.917 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:10.917+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:11.563 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:11.563+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:11.579 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:11.578+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:11.893 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:11.893+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:12.537 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:12.536+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:12.563 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:12.563+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:12.900 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:12.899+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:13.488 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:13.488+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:13.540 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:13.539+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:13.921 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:13.921+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:14.492 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:14.491+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:14.519 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:14.518+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:14.916 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:14.916+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:15.467 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:15.467+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:15.529 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:15.528+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:15.889 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:15.889+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:16.422 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:16.421+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:16.536 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:16.535+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:16.870 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:16.870+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:17.384 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:17.383+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:17.531 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:17.531+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:17.874 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:17.873+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:18.414 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:18.413+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:18.549 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:18.550+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:18.924 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:18.924+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:19.367 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:19.367+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:19.505 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:19.504+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:19.932 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:19.932+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:20.329 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:20.329+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:20.513 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:20.514+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:20.949 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:20.949+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:21.301 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:21.302+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:21.550 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:21.550+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:21.901 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:21.902+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:22.277 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:22.278+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:22.555 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:22.555+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:22.925 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:22.926+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:23.261 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:23.262+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:23.570 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:23.571+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:23.931 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:23.932+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:24.214 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:24.214+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:24.604 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:24.605+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:24.963 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:24.963+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:25.177 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:25.177+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:25.605 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:25.606+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:25.926 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:25.926+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:26.163 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:26.164+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:26.600 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:26.601+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:26.975 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:26.976+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:27.179 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:27.180+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:27.590 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:27.591+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:28.003 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:28.004+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:28.172 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:28.173+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:28.568 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:28.569+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:29.005 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:29.006+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:29.213 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:29.214+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:29.558 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:29.559+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:30.001 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:30.001+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:30.194 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:30.194+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:30.544 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:30.545+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:31.021 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:31.022+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:31.169 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:31.168+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:31.563 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:31.564+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:32.004 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:32.005+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:32.197 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:32.197+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:32.610 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:32.611+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:32.970 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:32.971+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:33.154 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:33.155+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:33.607 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:33.608+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:33.941 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:33.941+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:34.163 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:34.163+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:34.636 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:34.636+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:34.929 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:34.929+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:35.115 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:35.115+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:35.606 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:35.606+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:35.921 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:35.921+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:36.140 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:36.140+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:36.580 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:36.580+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:36.881 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:36.881+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:37.120 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:37.120+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:37.577 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:37.576+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:37.905 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:37.905+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:38.133 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:38.132+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:38.561 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:38.561+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:38.869 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:38.868+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:39.135 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:39.134+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:39.574 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:39.574+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:39.890 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:39.890+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:40.132 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:40.131+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:40.571 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:40.571+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:40.909 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:40.909+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:41.145 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:41.145+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:41.583 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:41.583+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:41.922 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:41.922+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:42.097 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:42.096+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:42.621 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:42.621+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:42.935 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:42.934+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:43.055 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:43.055+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:43.606 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:43.606+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:43.976 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:43.976+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:44.058 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:44.058+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:44.561 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:44.561+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:44.974 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:44.974+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:45.106 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:45.106+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:45.535 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:45.535+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:46.009 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:46.009+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:46.115 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:46.115+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:46.486 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:46.486+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:47.044 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:47.043+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:47.105 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:47.105+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:47.438 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:47.437+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:48.003 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:48.002+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:48.143 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:48.143+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:48.486 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:48.486+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:48.969 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:48.969+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:49.180 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:49.180+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:49.517 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:49.516+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:50.014 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:50.014+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:50.160 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:50.160+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:50.547 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:50.547+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:50.986 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:50.986+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:51.121 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:51.121+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:51.568 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:51.568+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:52.001 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:52.000+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:52.108 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:52.107+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:52.574 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:52.573+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:52.995 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:52.994+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:53.062 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:53.062+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:53.526 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:53.526+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:54.025 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:54.024+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:54.041 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:54.041+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:54.557 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:54.557+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:55.027 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:55.027+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:55.063 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:55.063+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:55.558 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:55.558+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:56.042 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:56.042+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:56.079 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:56.079+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:56.530 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:56.529+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:57.029 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:57.029+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:57.100 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:57.100+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:57.496 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:57.496+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:57.999 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:57.998+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:58.065 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:58.065+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:58.498 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:58.498+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:59.028 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:59.027+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:59.056 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:19:59.055+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:59.534 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:19:59.533+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:19:59.989 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:19:59.989+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:00.042 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:00.042+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:00.566 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:00.566+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:00.972 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:00.971+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:01.040 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:01.040+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:01.584 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:01.583+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:01.938 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:01.938+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:02.008 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:02.008+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:02.600 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:02.600+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:02.897 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:02.896+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:03.002 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:03.002+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:03.570 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:03.571+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:03.874 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:03.873+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:04.017 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:04.017+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:04.565 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:04.566+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:04.857 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:04.857+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:04.988 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:04.986+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:05.612 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:05.613+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:05.825 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:05.826+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:05.973 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:05.974+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:06.571 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:06.572+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:06.815 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:06.816+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:06.954 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:06.954+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:07.526 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:07.527+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:07.858 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:07.858+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:07.998 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:07.998+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:08.490 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:08.491+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:08.886 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:08.887+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:08.997 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:08.998+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:09.503 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:09.504+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:09.841 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:09.842+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:09.976 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:09.977+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:10.498 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:10.499+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:10.813 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:10.814+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:10.934 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:10.935+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:11.462 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:11.462+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:11.807 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:11.807+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:11.904 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:11.905+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:12.415 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:12.416+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:12.787 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:12.788+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:12.917 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:12.918+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:13.408 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:13.409+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:13.746 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:13.746+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:13.953 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:13.954+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:14.382 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:14.383+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:14.717 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:14.718+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:14.930 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:14.931+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:15.406 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:15.407+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:15.689 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:15.690+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:15.980 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:15.981+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:16.371 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:16.372+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:16.694 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:16.695+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:16.936 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:16.937+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:17.334 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:17.334+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:17.684 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:17.685+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:17.955 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:17.956+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:18.331 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:18.332+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:18.729 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:18.728+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:18.924 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:18.923+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:19.336 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:19.336+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:19.690 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:19.690+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:19.937 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:19.937+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:20.347 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:20.347+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:20.682 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:20.681+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:20.967 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:20.967+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:21.394 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:21.394+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:21.719 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:21.717+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:22.009 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:22.008+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:22.396 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:22.396+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:22.706 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:22.706+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:23.025 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:23.025+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:23.424 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:23.423+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:23.742 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:23.742+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:24.021 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:24.020+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:24.412 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:24.412+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:24.706 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:24.706+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:25.028 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:25.028+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:25.411 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:25.411+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:25.743 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:25.743+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:26.000 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:26.000+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:26.427 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:26.427+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:26.773 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:26.772+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:27.022 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:27.021+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:27.400 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:27.400+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:27.779 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:27.779+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:28.037 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:28.037+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:28.377 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:28.377+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:28.811 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:28.810+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:29.061 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:29.060+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:29.407 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:29.407+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:29.813 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:29.813+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:30.062 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:30.061+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:30.396 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:30.396+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:30.844 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:30.844+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:31.087 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:31.087+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:31.384 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:31.384+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:31.805 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:31.805+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:32.073 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:32.073+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:32.341 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:32.341+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:32.836 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:32.835+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:33.092 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:33.092+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:33.339 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:33.339+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:33.815 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:33.814+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:34.117 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:34.117+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:34.296 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:34.296+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:34.825 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:34.824+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:35.116 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:35.116+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:35.324 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:35.323+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:35.812 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:35.811+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:36.075 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:36.075+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:36.361 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:36.360+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:36.788 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:36.788+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:37.105 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:37.105+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:37.384 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:37.384+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:37.748 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:37.747+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:38.067 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:38.067+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:38.339 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:38.339+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:38.719 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:38.719+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:39.087 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:39.086+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:39.300 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:39.300+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:39.693 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:39.693+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:40.081 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:40.081+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:40.283 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:40.283+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:40.704 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:40.703+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:41.043 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:41.042+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:41.303 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:41.303+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:41.745 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:41.745+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:42.081 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:42.081+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:42.265 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:42.264+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:42.709 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:42.708+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:43.116 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:43.116+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:43.232 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:43.232+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:43.733 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:43.733+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:44.164 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:44.164+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:44.232 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:44.232+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:44.756 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:44.756+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:45.129 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:45.128+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:45.239 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:45.239+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:45.769 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:45.769+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:46.147 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:46.146+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:46.197 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:46.197+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:46.722 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:46.722+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:47.179 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:47.179+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:47.226 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:47.225+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:47.721 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:47.721+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:48.186 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:48.186+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:48.189 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:48.189+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:48.687 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:48.686+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:49.170 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:49.170+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:49.196 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:49.196+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:49.732 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:49.731+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:50.121 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:50.120+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:50.169 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:50.169+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:50.708 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:50.708+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:51.154 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:51.154+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:51.164 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:51.165+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:51.702 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:51.702+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:52.137 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:52.137+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:52.186 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:52.187+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:52.715 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:52.715+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:53.165 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:53.166+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:53.217 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:53.218+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:53.710 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:53.711+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:54.181 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:54.180+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:54.210 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:54.210+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:54.704 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:54.705+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:55.185 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:55.186+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:55.186 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:55.187+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:55.726 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:55.726+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:56.146 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:56.147+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:56.155 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:56.156+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:56.726 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:56.727+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:57.127 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:57.128+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:57.153 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:57.154+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:57.695 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:57.696+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:58.107 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:58.108+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:58.119 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:58.120+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:58.668 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:58.668+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:59.062 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:20:59.063+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:59.092 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:20:59.093+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:20:59.712 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:20:59.713+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:00.032 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:00.033+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:00.056 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:00.057+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:00.723 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:00.723+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:01.030 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:01.031+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:01.057 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:01.058+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:01.696 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:01.697+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:02.039 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:02.040+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:02.046 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:02.047+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:02.703 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:02.704+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:03.074 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:03.075+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:03.082 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:03.083+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:03.718 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:03.718+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:04.038 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:04.038+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:04.105 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:04.105+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:04.713 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:04.713+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:05.040 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:05.040+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:05.086 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:05.086+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:05.719 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:05.719+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:06.046 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:06.046+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:06.078 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:06.078+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:06.736 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:06.736+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:07.041 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:07.041+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:07.052 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:07.052+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:07.720 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:07.719+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:08.005 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:08.005+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:08.088 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:08.088+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:08.766 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:08.766+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:08.982 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:08.982+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:09.058 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:09.058+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:09.720 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:09.720+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:09.995 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:09.994+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:10.096 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:10.096+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:10.740 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:10.740+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:10.993 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:10.993+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:11.076 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:11.076+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:11.760 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:11.760+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:11.975 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:11.975+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:12.080 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:12.079+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:12.745 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:12.745+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:13.004 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:13.004+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:13.047 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:13.047+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:13.718 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:13.718+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:14.020 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:14.020+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:14.040 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:14.040+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:14.699 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:14.698+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:15.000 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:15.000+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:15.050 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:15.049+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:15.674 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:15.673+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:16.006 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:16.005+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:16.018 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:16.018+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:16.652 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:16.652+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:17.025 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:17.025+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:17.042 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:17.042+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:17.634 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:17.633+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:18.003 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:18.002+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:18.054 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:18.054+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:18.594 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:18.594+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:19.030 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:19.030+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:19.043 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:19.042+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:19.613 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:19.612+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:20.065 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:20.065+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:20.066 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:20.066+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:20.604 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:20.603+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:21.034 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:21.034+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:21.064 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:21.064+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:21.616 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:21.616+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:21.986 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:21.985+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:22.050 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:22.050+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:22.596 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:22.596+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:22.959 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:22.959+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:23.041 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:23.040+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:23.624 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:23.624+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:23.914 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:23.914+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:24.011 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:24.011+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:24.595 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:24.595+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:24.947 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:24.946+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:25.027 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:25.027+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:25.564 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:25.562+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:25.964 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:25.964+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:25.983 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:25.983+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:26.567 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:26.566+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:27.000 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:26.999+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:27.010 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:27.010+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:27.540 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:27.540+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:28.042 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:28.042+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:28.050 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:28.049+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:28.564 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:28.563+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:29.074 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:29.074+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:29.091 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:29.091+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:29.546 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:29.545+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:30.081 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:30.079+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:30.104 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:30.104+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:30.584 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:30.583+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:31.068 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:31.068+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:31.116 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:31.116+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:31.628 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:31.627+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:32.068 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:32.068+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:32.146 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:32.145+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:32.586 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:32.586+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:33.076 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:33.076+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:33.169 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:33.169+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:33.553 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:33.552+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:34.098 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:34.097+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:34.140 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:34.141+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:34.556 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:34.556+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:35.106 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:35.106+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:35.139 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:35.139+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:35.591 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:35.592+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:36.114 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:36.115+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:36.143 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:36.144+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:36.583 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:36.584+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:37.076 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:37.076+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:37.164 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:37.164+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:37.560 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:37.561+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:38.055 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:38.055+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:38.124 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:38.125+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:38.584 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:38.585+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:39.057 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:39.057+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:39.115 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:39.116+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:39.630 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:39.630+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:40.057 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:40.058+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:40.114 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:40.115+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:40.609 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:40.610+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:41.016 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:41.017+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:41.112 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:41.113+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:41.594 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:41.595+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:42.037 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:42.038+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:42.100 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:42.100+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:42.635 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:42.636+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:43.025 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:43.026+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:43.146 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:43.146+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:43.637 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:43.638+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:44.031 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:44.032+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:44.152 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:44.152+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:44.613 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:44.614+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:45.019 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:45.020+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:45.199 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:45.200+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:45.639 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:45.640+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:46.070 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:46.071+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:46.197 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:46.198+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:46.592 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:46.593+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:47.034 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:47.035+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:47.219 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:47.219+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:47.580 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:47.581+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:48.007 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:48.008+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:48.183 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:48.184+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:48.555 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:48.554+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:48.974 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:48.973+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:49.174 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:49.174+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:49.545 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:49.545+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:49.990 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:49.990+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:50.131 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:50.131+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:50.504 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:50.503+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:51.027 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:51.026+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:51.151 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:51.150+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:51.502 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:51.502+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:52.049 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:52.049+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:52.163 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:52.162+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:52.495 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:52.495+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:53.073 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:53.073+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:53.164 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:53.164+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:53.516 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:53.515+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:54.102 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:54.102+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:54.153 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:54.153+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:54.541 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:54.541+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:55.100 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:55.100+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:55.160 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:55.160+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:55.499 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:55.498+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:56.121 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:56.121+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:56.145 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:56.145+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:56.484 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:56.484+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:57.097 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:57.097+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:57.099 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:57.099+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:57.525 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:57.525+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:58.085 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:58.085+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:58.127 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:58.127+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:58.529 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:58.529+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:59.038 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:21:59.037+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:59.152 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:21:59.152+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:21:59.529 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:21:59.529+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:00.003 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:00.003+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:00.179 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:00.179+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:00.527 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:00.527+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:00.995 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:00.995+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:01.177 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:01.177+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:01.532 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:01.532+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:01.963 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:01.962+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:02.162 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:02.162+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:02.563 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:02.563+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:02.951 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:02.951+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:03.115 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:03.114+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:03.526 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:03.526+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:03.618 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T19:22:03.618+0000 7fccf53da700 0 -- 172.21.15.87:0/3193322219 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7fccd803a500 msgr2=0x7fccd804bd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-24T19:22:03.990 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:03.990+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:04.134 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:04.134+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:04.535 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:04.535+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:04.976 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:04.976+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:05.102 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:05.102+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:05.547 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:05.546+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:05.999 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:05.999+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:06.085 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:06.084+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:06.595 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:06.595+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:06.986 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:06.986+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:07.058 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:07.058+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:07.598 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:07.598+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:07.940 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:07.940+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:08.022 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:08.022+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:08.584 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:08.584+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:08.901 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:08.901+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:09.066 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:09.066+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:09.579 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:09.579+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:09.932 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:09.932+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:10.068 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:10.068+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:10.549 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:10.549+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:10.939 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:10.938+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:11.035 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:11.035+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:11.543 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:11.543+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:11.972 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:11.971+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:12.038 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:12.038+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:12.559 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:12.558+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:12.940 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:12.940+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:13.082 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:13.081+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:13.603 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:13.602+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:13.988 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:13.988+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:14.042 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:14.041+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:14.593 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:14.593+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:14.943 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:14.942+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:14.996 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:14.996+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:15.554 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:15.553+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:15.957 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:15.957+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:15.989 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:15.989+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:16.583 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:16.582+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:16.912 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:16.912+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:16.994 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:16.993+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:17.539 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:17.539+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:17.889 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:17.889+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:17.983 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:17.982+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:18.507 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:18.507+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:18.915 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:18.915+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:18.946 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:18.946+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:19.499 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:19.498+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:19.920 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:19.920+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:19.953 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:19.952+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:20.468 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:20.469+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:20.873 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:20.874+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:20.954 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:20.955+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:21.429 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:21.430+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:21.910 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:21.910+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:22.003 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:22.004+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:22.467 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:22.468+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:22.933 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:22.934+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:22.958 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:22.959+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:23.438 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:23.439+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:23.887 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:23.887+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:23.988 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:23.989+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:24.478 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:24.479+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:24.903 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:24.904+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:24.998 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:24.999+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:25.488 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:25.489+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:25.916 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:25.917+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:26.021 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:26.022+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:26.450 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:26.451+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:26.959 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:26.960+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:26.981 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:26.982+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:27.468 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:27.468+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:27.934 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:27.935+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:27.980 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:27.980+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:28.450 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:28.450+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:28.897 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:28.898+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:28.982 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:28.982+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:29.442 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:29.442+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:29.896 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:29.897+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:30.030 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:30.031+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:30.408 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:30.408+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:30.855 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:30.854+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:31.021 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:31.021+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:31.458 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:31.459+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:31.841 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:31.842+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:32.058 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:32.059+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:32.478 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:32.479+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:32.845 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:32.846+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:33.100 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:33.100+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:33.516 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:33.516+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:33.822 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:33.822+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:34.126 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:34.126+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:34.551 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:34.550+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:34.811 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:34.810+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:35.150 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:35.149+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:35.516 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:35.516+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:35.849 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:35.849+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:36.113 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:36.113+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:36.521 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:36.521+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:36.877 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:36.876+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:37.064 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:37.064+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:37.482 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:37.481+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:37.904 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:37.903+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:38.025 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:38.025+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:38.498 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:38.497+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:38.888 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:38.888+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:39.000 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:39.000+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:39.471 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:39.471+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:39.874 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:39.873+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:40.040 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:40.040+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:40.504 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:40.504+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:40.880 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:40.880+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:41.080 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:41.079+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:41.497 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:41.497+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:41.833 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:41.831+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:42.044 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:42.044+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:42.475 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:42.475+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:42.877 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:42.877+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:43.002 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:43.002+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:43.444 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:43.444+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:43.915 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:43.915+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:43.993 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:43.993+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:44.485 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:44.485+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:44.891 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:44.891+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:44.957 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:44.957+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:45.501 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:45.501+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:45.916 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:45.916+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:45.953 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:45.953+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:46.477 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:46.477+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:46.952 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:46.951+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:46.990 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:46.990+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:47.464 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:47.464+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:47.944 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:47.944+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:47.964 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:47.964+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:48.422 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:48.422+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:48.947 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:48.947+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:49.007 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:49.007+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:49.436 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:49.436+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:49.952 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:49.952+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:50.027 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:50.027+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:50.408 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:50.408+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:50.913 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:50.912+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:51.052 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:51.050+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:51.432 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:51.432+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:51.897 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:51.897+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:52.037 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:52.036+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:52.469 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:52.468+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:52.892 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:52.892+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:53.018 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:53.018+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:53.474 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:53.474+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:53.893 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:53.892+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:53.990 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:53.990+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:54.483 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:54.482+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:54.888 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:54.888+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:54.954 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:54.954+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:55.495 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:55.495+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:55.879 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:55.879+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:56.000 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:56.000+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:56.477 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:56.476+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:56.902 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:56.902+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:56.971 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:56.971+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:57.442 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:57.442+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:57.899 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:57.899+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:58.012 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:58.012+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:58.456 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:58.456+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:58.854 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:58.854+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:59.002 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:22:59.002+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:59.459 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:22:59.458+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:22:59.870 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:22:59.870+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:00.033 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:00.033+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:00.434 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:00.434+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:00.884 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:00.884+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:01.026 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:01.026+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:01.386 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:01.386+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:01.877 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:01.877+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:02.059 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:02.057+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:02.354 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:02.354+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:02.843 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:02.843+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:03.078 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:03.079+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:03.397 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:03.396+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:03.870 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:03.870+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:04.045 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:04.045+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:04.390 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:04.390+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:04.905 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:04.906+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:04.999 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:05.000+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:05.406 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:05.407+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:05.947 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:05.948+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:06.015 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:06.016+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:06.416 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:06.417+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:06.911 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:06.912+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:07.063 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:07.064+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:07.459 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:07.460+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:07.947 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:07.948+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:08.052 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:08.053+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:08.422 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:08.423+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:08.995 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:08.996+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:09.059 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:09.060+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:09.398 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:09.398+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:10.035 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:10.036+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:10.041 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:10.042+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:10.375 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:10.376+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:10.998 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:10.999+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:11.063 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:11.063+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:11.361 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:11.361+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:12.009 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:12.010+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:12.064 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:12.065+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:12.407 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:12.408+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:13.039 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:13.040+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:13.102 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:13.099+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:13.379 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:13.379+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:14.021 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:14.022+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:14.146 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:14.147+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:14.411 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:14.411+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:15.053 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:15.053+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:15.101 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:15.101+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:15.439 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:15.439+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:16.061 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:16.062+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:16.099 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:16.100+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:16.425 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:16.426+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:17.022 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:17.022+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:17.118 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:17.119+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:17.407 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:17.408+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:18.056 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:18.056+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:18.119 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:18.119+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:18.398 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:18.397+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:19.104 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:19.104+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:19.118 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:19.118+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:19.383 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:19.383+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:20.066 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:20.066+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:20.161 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:20.161+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:20.346 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:20.346+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:21.093 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:21.093+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:21.178 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:21.178+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:21.389 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:21.389+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:22.054 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:22.053+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:22.208 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:22.208+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:22.397 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:22.396+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:23.074 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:23.074+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:23.178 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:23.178+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:23.385 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:23.385+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:24.086 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:24.086+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:24.178 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:24.178+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:24.347 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:24.347+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:25.113 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:25.112+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:25.153 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:25.153+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:25.326 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:25.326+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:26.114 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:26.114+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:26.182 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:26.182+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:26.284 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:26.284+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:27.127 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:27.127+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:27.164 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:27.164+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:27.251 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:27.250+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:28.143 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:28.142+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:28.186 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:28.185+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:28.290 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:28.290+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:29.104 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:29.104+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:29.162 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:29.161+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:29.241 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:29.241+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:30.137 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:30.137+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:30.148 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:30.148+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:30.290 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:30.289+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:31.135 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:31.134+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:31.152 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:31.152+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:31.303 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:31.302+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:32.094 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:32.094+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:32.137 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:32.137+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:32.304 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:32.304+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:33.068 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:33.067+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:33.137 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:33.137+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:33.345 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:33.345+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:34.038 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:34.038+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:34.127 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:34.127+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:34.369 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:34.369+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:35.042 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:35.042+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:35.139 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:35.138+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:35.327 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:35.327+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:36.075 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:36.075+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:36.127 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:36.127+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:36.373 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:36.373+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:37.096 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:37.096+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:37.103 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:37.102+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:37.370 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:37.370+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:38.098 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:38.098+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:38.118 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:38.118+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:38.400 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:38.400+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:39.090 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:39.090+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:39.094 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:39.094+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:39.408 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:39.408+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:40.114 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:40.114+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:40.141 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:40.141+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:40.369 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:40.369+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:41.134 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:41.134+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:41.148 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:41.145+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:41.356 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:41.354+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:42.103 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:42.103+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:42.167 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:42.167+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:42.402 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:42.402+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:43.054 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:43.053+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:43.128 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:43.128+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:43.439 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:43.439+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:44.078 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:44.078+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:44.110 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:44.110+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:44.481 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:44.480+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:45.065 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:45.065+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:45.107 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:45.106+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:45.497 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:45.497+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:46.041 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:46.041+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:46.059 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:46.059+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:46.474 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:46.474+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:47.029 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:47.029+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:47.042 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:47.042+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:47.459 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:47.459+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:48.033 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:48.033+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:48.034 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:48.035+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:48.480 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:48.480+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:49.069 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:49.069+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:49.079 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:49.078+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:49.520 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:49.520+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:50.032 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:50.033+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:50.033 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:50.034+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:50.549 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:50.550+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:50.990 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:50.991+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:51.052 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:51.053+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:51.501 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:51.502+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:51.985 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:51.986+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:52.075 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:52.076+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:52.458 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:52.459+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:52.997 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:52.997+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:53.065 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:53.066+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:53.505 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:53.506+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:54.035 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:54.036+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:54.070 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:54.071+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:54.555 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:54.555+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:55.032 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:55.032+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:55.108 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:55.108+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:55.538 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:55.538+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:55.996 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:55.996+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:56.132 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:56.132+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:56.587 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:56.587+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:56.973 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:56.973+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:57.143 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:57.144+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:57.592 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:57.593+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:57.969 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:57.970+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:58.134 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:58.135+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:58.637 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:58.637+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:58.945 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:58.946+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:59.088 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:23:59.089+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:59.641 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:23:59.641+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:23:59.955 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:23:59.955+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:00.055 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:00.056+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:00.685 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:00.686+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:00.957 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:00.957+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:01.102 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:01.102+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:01.662 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:01.663+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:01.924 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:01.925+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:02.141 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:02.140+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:02.639 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:02.639+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:02.879 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:02.879+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:03.142 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:03.142+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:03.635 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:03.635+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:03.905 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:03.905+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:04.185 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:04.185+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:04.661 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:04.660+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:04.905 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:04.905+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:05.136 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:05.136+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:05.647 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:05.647+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:05.893 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:05.892+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:06.137 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:06.137+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:06.612 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:06.611+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:06.885 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:06.885+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:07.142 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:07.142+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:07.573 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:07.573+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:07.875 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:07.875+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:08.152 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:08.152+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:08.546 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:08.546+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:08.887 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:08.887+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:09.154 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:09.153+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:09.532 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:09.531+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:09.939 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:09.927+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:10.200 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:10.199+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:10.557 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:10.556+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:10.944 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:10.943+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:11.239 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:11.238+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:11.518 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:11.517+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:11.968 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:11.966+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:12.261 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:12.260+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:12.561 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:12.560+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:13.009 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:13.009+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:13.286 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:13.286+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:13.601 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:13.601+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:14.017 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:14.017+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:14.281 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:14.281+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:14.638 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:14.638+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:15.028 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:15.028+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:15.293 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:15.292+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:15.689 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:15.688+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:15.983 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:15.983+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:16.323 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:16.323+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:16.682 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:16.681+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:16.982 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:16.982+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:17.330 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:17.330+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:17.728 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:17.728+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:18.013 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:18.013+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:18.369 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:18.368+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:18.706 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:18.706+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:19.035 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:19.035+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:19.338 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:19.338+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:19.742 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:19.742+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:20.058 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:20.058+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:20.323 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:20.323+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:20.702 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:20.702+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:21.102 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:21.102+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:21.366 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:21.366+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:21.708 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:21.708+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:22.069 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:22.069+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:22.362 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:22.361+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:22.751 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:22.751+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:23.117 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:23.117+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:23.394 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:23.394+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:23.781 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:23.781+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:24.165 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:24.165+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:24.425 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:24.425+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:24.737 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:24.737+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:25.169 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:25.168+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:25.392 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:25.392+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:25.697 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:25.697+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:26.154 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:26.153+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:26.353 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:26.353+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:26.699 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:26.699+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:27.114 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:27.114+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:27.399 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:27.399+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:27.712 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:27.712+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:28.161 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:28.160+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:28.441 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:28.441+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:28.728 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:28.728+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:29.120 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:29.120+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:29.446 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:29.445+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:29.720 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:29.720+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:30.090 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:30.090+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:30.447 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:30.447+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:30.684 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:30.683+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:31.101 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:31.101+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:31.409 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:31.409+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:31.635 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:31.635+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:32.138 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:32.138+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:32.405 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:32.405+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:32.636 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:32.636+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:33.177 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:33.177+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:33.358 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:33.359+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:33.628 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:33.628+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:34.162 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:34.162+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:34.384 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:34.384+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:34.657 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:34.658+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:35.162 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:35.162+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:35.402 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:35.403+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:35.657 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:35.658+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:36.130 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:36.131+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:36.363 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:36.364+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:36.703 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:36.704+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:37.170 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:37.171+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:37.405 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:37.406+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:37.684 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:37.685+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:38.169 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:38.170+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:38.380 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:38.381+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:38.638 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:38.639+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:39.122 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:39.123+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:39.336 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:39.337+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:39.661 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:39.661+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:40.153 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:40.154+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:40.343 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:40.344+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:40.652 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:40.650+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:41.158 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:41.159+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:41.389 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:41.389+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:41.620 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:41.620+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:42.164 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:42.165+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:42.360 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:42.361+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:42.577 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:42.578+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:43.156 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:43.156+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:43.357 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:43.357+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:43.534 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:43.534+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:44.175 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:44.176+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:44.359 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:44.360+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:44.527 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:44.528+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:45.190 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:45.191+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:45.324 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:45.325+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:45.510 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:45.511+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:46.238 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:46.235+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:46.342 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:46.343+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:46.481 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:46.482+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:47.272 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:47.272+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:47.349 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:47.349+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:47.473 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:47.473+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:48.307 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:48.307+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:48.312 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:48.312+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:48.456 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:48.456+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:49.272 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:49.272+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:49.362 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:49.361+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:49.441 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:49.440+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:50.232 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:50.232+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:50.336 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:50.336+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:50.410 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:50.410+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:51.238 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:51.238+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:51.297 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:51.297+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:51.385 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:51.385+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:52.260 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:52.260+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:52.277 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:52.277+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:52.376 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:52.376+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:53.272 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:53.272+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:53.288 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:53.287+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:53.410 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:53.410+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:54.245 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:54.245+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:54.305 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:54.305+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:54.409 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:54.408+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:55.285 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:55.284+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:55.332 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:55.331+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:55.433 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:55.433+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:56.296 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:56.294+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:56.310 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:56.310+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:56.419 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:56.419+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:57.264 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:57.264+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:57.286 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:57.286+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:57.466 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:57.465+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:58.251 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:58.251+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:58.311 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:58.310+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:58.492 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:58.491+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:59.226 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:24:59.226+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:59.291 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:24:59.291+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:24:59.523 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:24:59.523+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:00.210 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:00.210+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:00.319 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:00.318+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:00.478 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:00.478+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:01.256 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:01.255+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:01.344 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:01.344+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:01.513 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:01.513+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:02.274 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:02.274+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:02.323 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:02.322+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:02.522 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:02.522+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:03.272 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:03.272+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:03.366 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:03.366+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:03.544 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:03.544+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:04.248 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:04.248+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:04.334 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:04.334+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:04.518 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:04.517+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:05.272 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:05.272+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:05.326 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:05.326+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:05.475 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:05.475+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:06.261 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:06.260+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:06.285 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:06.285+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:06.466 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:06.466+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:07.268 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:07.267+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:07.277 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:07.277+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:07.494 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:07.494+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:08.236 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:08.236+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:08.286 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:08.286+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:08.447 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:08.447+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:09.252 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:09.252+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:09.253 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:09.253+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:09.457 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:09.457+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:10.218 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:10.218+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:10.242 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:10.242+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:10.481 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:10.480+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:11.240 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:11.239+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:11.278 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:11.278+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:11.502 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:11.502+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:12.208 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:12.208+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:12.262 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:12.262+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:12.472 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:12.472+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:13.215 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:13.215+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:13.251 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:13.251+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:13.469 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:13.468+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:14.194 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:14.194+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:14.212 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:14.211+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:14.447 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:14.447+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:15.164 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:15.164+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:15.184 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:15.184+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:15.491 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:15.491+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:16.143 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:16.143+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:16.210 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:16.210+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:16.460 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:16.460+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:17.101 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:17.100+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:17.209 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:17.209+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:17.418 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:17.418+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:18.063 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:18.063+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:18.203 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:18.203+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:18.441 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:18.442+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:19.077 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:19.077+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:19.168 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:19.169+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:19.464 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:19.465+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:20.065 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:20.065+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:20.135 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:20.135+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:20.451 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:20.452+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:21.049 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:21.050+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:21.127 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:21.128+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:21.502 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:21.502+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:22.014 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:22.015+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:22.129 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:22.129+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:22.545 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:22.546+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:22.984 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:22.985+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:23.106 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:23.106+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:23.593 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:23.593+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:23.986 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:23.987+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:24.059 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:24.060+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:24.553 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:24.554+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:25.028 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:25.029+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:25.109 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:25.110+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:25.556 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:25.557+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:26.035 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:26.036+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:26.070 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:26.071+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:26.514 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:26.515+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:27.025 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:27.025+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:27.085 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:27.085+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:27.483 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:27.484+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:28.025 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:28.026+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:28.078 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:28.079+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:28.455 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:28.456+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:28.989 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:28.990+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:29.086 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:29.087+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:29.474 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:29.475+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:29.958 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:29.959+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:30.067 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:30.068+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:30.467 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:30.468+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:30.910 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:30.911+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:31.067 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:31.068+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:31.429 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:31.429+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:31.872 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:31.872+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:32.032 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:32.032+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:32.396 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:32.396+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:32.846 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:32.846+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:33.025 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:33.025+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:33.378 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:33.377+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:33.850 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:33.850+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:34.003 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:34.003+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:34.343 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:34.343+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:34.846 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:34.846+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:34.977 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:34.977+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:35.302 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:35.302+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:35.883 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:35.883+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:35.944 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:35.944+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:36.259 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:36.259+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:36.863 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:36.863+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:36.925 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:36.925+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:37.237 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:37.237+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:37.826 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:37.826+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:37.937 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:37.937+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:38.255 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:38.254+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:38.797 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:38.794+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:38.969 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:38.968+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:39.223 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:39.223+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:39.756 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:39.756+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:39.983 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:39.982+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:40.255 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:40.255+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:40.758 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:40.758+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:41.007 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:41.007+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:41.212 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:41.212+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:41.720 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:41.719+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:42.007 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:42.007+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:42.189 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:42.188+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:42.769 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:42.768+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:42.973 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:42.972+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:43.140 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:43.140+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:43.726 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:43.726+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:43.994 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:43.993+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:44.142 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:44.141+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:44.744 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:44.743+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:45.006 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:45.005+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:45.141 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:45.141+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:45.718 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:45.717+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:45.996 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:45.996+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:46.187 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:46.187+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:46.760 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:46.759+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:47.011 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:47.010+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:47.172 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:47.171+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:47.784 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:47.783+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:48.005 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:48.005+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:48.216 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:48.216+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:48.771 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:48.771+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:49.035 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:49.035+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:49.209 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:49.208+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:49.751 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:49.751+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:50.034 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:50.034+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:50.188 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:50.188+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:50.732 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:50.731+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:51.061 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:51.060+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:51.192 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:51.192+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:51.766 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:51.765+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:52.071 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:52.071+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:52.237 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:52.237+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:52.794 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:52.794+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:53.031 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:53.031+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:53.284 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:53.284+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:53.750 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:53.749+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:54.063 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:54.063+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:54.308 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:54.308+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:54.776 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:54.775+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:55.022 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:55.021+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:55.292 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:55.292+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:55.739 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:55.739+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:56.003 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:56.003+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:56.302 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:56.301+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:56.711 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:56.711+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:57.003 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:57.003+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:57.294 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:57.293+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:57.735 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:57.734+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:58.007 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:58.007+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:58.309 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:58.309+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:58.768 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:58.768+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:58.965 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:58.964+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:59.352 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:25:59.352+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:59.805 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:25:59.805+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:25:59.937 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:25:59.936+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:00.402 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:00.402+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:00.804 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:00.804+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:00.890 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:00.890+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:01.437 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:01.436+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:01.819 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:01.819+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:01.894 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:01.894+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:02.435 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:02.434+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:02.818 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:02.818+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:02.927 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:02.927+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:03.421 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:03.422+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:03.778 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:03.778+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:03.894 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:03.895+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:04.443 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:04.444+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:04.816 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:04.817+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:04.878 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:04.878+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:05.463 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:05.463+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:05.787 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:05.788+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:05.913 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:05.913+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:06.475 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:06.475+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:06.817 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:06.818+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:06.955 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:06.955+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:07.499 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:07.500+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:07.803 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:07.803+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:07.996 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:07.997+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:08.491 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:08.491+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:08.831 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:08.831+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:08.971 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:08.971+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:09.128 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T19:26:09.129+0000 7fccf4bd9700 0 -- 172.21.15.87:0/3193322219 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fccd8033f40 msgr2=0x7fccd803cf50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-24T19:26:09.454 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:09.455+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:09.797 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:09.798+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:09.984 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:09.985+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:10.482 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:10.483+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:10.824 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:10.825+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:11.019 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:11.020+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:11.435 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:11.436+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:11.843 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:11.844+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:12.017 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:12.018+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:12.398 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:12.399+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:12.849 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:12.850+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:12.992 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:12.993+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:13.356 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:13.356+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:13.835 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:13.836+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:14.035 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:14.036+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:14.316 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:14.316+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:14.811 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:14.812+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:15.028 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:15.029+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:15.349 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:15.350+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:15.812 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:15.813+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:16.071 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:16.071+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:16.344 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:16.344+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:16.825 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:16.825+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:17.033 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:17.032+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:17.314 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:17.314+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:17.822 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:17.822+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:18.064 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:18.064+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:18.319 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:18.318+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:18.807 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:18.806+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:19.072 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:19.072+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:19.358 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:19.358+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:19.803 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:19.803+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:20.026 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:20.026+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:20.347 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:20.347+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:20.807 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:20.807+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:21.007 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:21.006+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:21.319 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:21.318+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:21.825 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:21.825+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:22.031 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:22.031+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:22.278 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:22.278+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:22.790 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:22.790+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:22.984 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:22.983+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:23.238 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:23.238+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:23.822 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:23.822+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:23.983 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:23.983+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:24.258 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:24.258+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:24.826 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:24.826+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:24.937 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:24.937+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:25.260 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:25.260+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:25.820 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:25.820+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:25.906 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:25.906+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:26.270 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:26.269+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:26.869 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:26.869+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:26.923 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:26.922+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:27.253 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:27.253+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:27.846 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:27.845+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:27.951 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:27.950+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:28.207 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:28.207+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:28.822 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:28.821+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:28.946 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:28.946+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:29.257 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:29.257+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:29.859 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:29.859+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:29.932 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:29.932+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:30.293 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:30.293+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:30.848 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:30.848+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:30.891 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:30.891+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:31.266 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:31.266+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:31.831 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:31.831+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:31.921 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:31.920+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:32.223 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:32.222+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:32.858 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:32.858+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:32.893 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:32.893+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:33.174 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:33.174+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:33.860 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:33.860+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:33.888 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:33.888+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:34.142 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:34.142+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:34.874 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:34.874+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:34.898 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:34.898+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:35.118 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:35.118+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:35.856 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:35.856+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:35.897 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:35.896+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:36.121 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:36.121+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:36.873 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:36.873+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:36.908 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:36.908+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:37.128 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:37.128+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:37.893 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:37.893+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:37.955 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:37.955+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:38.142 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:38.142+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:38.943 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:38.943+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:38.946 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:38.946+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:39.185 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:39.185+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:39.901 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:39.900+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:39.959 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:39.959+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:40.217 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:40.216+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:40.855 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:40.855+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:40.995 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:40.995+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:41.188 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:41.188+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:41.902 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:41.902+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:41.962 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:41.962+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:42.232 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:42.231+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:42.858 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:42.858+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:42.994 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:42.993+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:43.194 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:43.193+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:43.840 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:43.840+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:44.009 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:44.009+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:44.153 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:44.153+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:44.887 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:44.886+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:44.996 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:44.995+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:45.111 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:45.110+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:45.861 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:45.861+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:45.977 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:45.977+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:46.064 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:46.064+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:46.881 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:46.881+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:46.956 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:46.956+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:47.106 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:47.106+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:47.904 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:47.904+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:47.986 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:47.986+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:48.093 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:48.094+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:48.868 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:48.868+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:48.979 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:48.979+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:49.049 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:49.049+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:49.908 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:49.909+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:49.957 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:49.958+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:50.020 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:50.021+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:50.917 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:50.918+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:50.941 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:50.942+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:51.036 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:51.037+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:51.890 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:51.891+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:51.974 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:51.975+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:52.066 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:52.066+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:52.894 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:52.895+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:52.952 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:52.952+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:53.115 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:53.116+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:53.897 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:53.897+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:53.904 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:53.905+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:54.118 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:54.119+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:54.884 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:54.885+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:54.936 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:54.937+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:55.075 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:55.076+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:55.868 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:55.868+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:55.948 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:55.948+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:56.058 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:56.058+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:56.874 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:56.875+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:56.980 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:56.981+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:57.094 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:57.095+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:57.924 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:57.924+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:57.943 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:57.944+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:58.133 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:58.134+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:58.919 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:58.920+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:58.964 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:58.965+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:59.126 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:26:59.127+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:59.877 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:26:59.878+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:26:59.936 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:26:59.936+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:00.172 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:00.173+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:00.891 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:00.891+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:00.974 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:00.974+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:01.175 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:01.174+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:01.885 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:01.885+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:01.997 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:01.997+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:02.211 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:02.211+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:02.882 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:02.882+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:02.958 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:02.957+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:03.163 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:03.163+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:03.870 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:03.870+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:03.972 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:03.972+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:04.167 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:04.167+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:04.827 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:04.827+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:04.995 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:04.995+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:05.181 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:05.181+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:05.781 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:05.781+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:05.986 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:05.986+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:06.183 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:06.183+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:06.810 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:06.810+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:06.989 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:06.988+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:07.162 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:07.161+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:07.793 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:07.793+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:08.015 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:08.015+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:08.134 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:08.134+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:08.815 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:08.815+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:09.020 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:09.020+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:09.185 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:09.185+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:09.813 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:09.813+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:09.990 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:09.990+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:10.146 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:10.146+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:10.835 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:10.835+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:10.989 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:10.988+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:11.169 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:11.169+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:11.803 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:11.803+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:11.998 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:11.998+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:12.210 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:12.210+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:12.842 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:12.842+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:12.985 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:12.985+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:13.250 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:13.250+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:13.793 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:13.793+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:13.981 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:13.981+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:14.231 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:14.230+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:14.779 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:14.779+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:15.024 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:15.024+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:15.213 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:15.213+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:15.814 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:15.814+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:16.047 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:16.047+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:16.178 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:16.177+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:16.839 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:16.839+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:17.000 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:17.000+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:17.132 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:17.131+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:17.812 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:17.811+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:18.045 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:18.045+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:18.102 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:18.102+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:18.781 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:18.781+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:19.005 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:19.005+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:19.063 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:19.062+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:19.772 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:19.772+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:19.995 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:19.995+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:20.051 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:20.050+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:20.739 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:20.739+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:20.966 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:20.966+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:21.079 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:21.078+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:21.740 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:21.740+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:21.925 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:21.924+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:22.103 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:22.102+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:22.783 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:22.782+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:22.899 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:22.898+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:23.107 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:23.107+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:23.735 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:23.735+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:23.906 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:23.905+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:24.126 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:24.126+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:24.710 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:24.710+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:24.921 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:24.921+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:25.157 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:25.156+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:25.683 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:25.683+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:25.874 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:25.874+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:26.163 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:26.162+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:26.696 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:26.696+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:26.850 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:26.850+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:27.139 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:27.139+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:27.676 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:27.676+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:27.859 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:27.859+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:28.153 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:28.153+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:28.709 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:28.709+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:28.865 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:28.865+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:29.116 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:29.116+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:29.660 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:29.660+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:29.843 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:29.843+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:30.084 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:30.084+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:30.707 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:30.708+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:30.815 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:30.814+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:31.038 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:31.038+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:31.698 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:31.697+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:31.767 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:31.767+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:32.016 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:32.016+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:32.692 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:32.690+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:32.803 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:32.804+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:33.007 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:33.008+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:33.720 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:33.721+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:33.826 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:33.827+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:34.013 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:34.014+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:34.673 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:34.674+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:34.854 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:34.855+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:34.984 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:34.984+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:35.633 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:35.633+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:35.811 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:35.812+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:36.031 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:36.032+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:36.590 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:36.590+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:36.767 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:36.768+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:36.999 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:37.000+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:37.570 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:37.570+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:37.736 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:37.737+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:37.961 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:37.962+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:38.563 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:38.564+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:38.745 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:38.745+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:38.940 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:38.940+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:39.538 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:39.538+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:39.740 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:39.741+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:39.951 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:39.952+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:40.508 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:40.508+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:40.747 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:40.747+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:40.939 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:40.940+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:41.555 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:41.556+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:41.755 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:41.755+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:41.959 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:41.959+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:42.602 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:42.602+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:42.765 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:42.766+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:43.003 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:43.004+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:43.582 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:43.583+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:43.791 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:43.792+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:43.988 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:43.989+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:44.584 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:44.584+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:44.814 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:44.814+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:44.964 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:44.964+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:45.608 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:45.607+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:45.827 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:45.827+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:46.011 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:46.011+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:46.654 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:46.654+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:46.849 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:46.849+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:46.977 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:46.976+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:47.701 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:47.700+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:47.832 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:47.832+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:48.026 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:48.026+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:48.743 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:48.743+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:48.790 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:48.790+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:49.006 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:49.005+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:49.751 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:49.751+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:49.766 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:49.766+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:50.028 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:50.028+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:50.707 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:50.707+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:50.718 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:50.718+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:51.028 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:51.028+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:51.678 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:51.678+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:51.703 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:51.702+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:52.003 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:52.003+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:52.634 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:52.634+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:52.686 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:52.685+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:53.024 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:53.023+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:53.602 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:53.601+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:53.729 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:53.729+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:54.028 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:54.028+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:54.570 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:54.570+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:54.750 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:54.749+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:55.037 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:55.037+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:55.556 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:55.556+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:55.799 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:55.798+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:56.077 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:56.076+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:56.602 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:56.601+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:56.765 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:56.765+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:57.058 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:57.058+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:57.647 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:57.647+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:57.779 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:57.778+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:58.101 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:58.101+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:58.695 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:58.695+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:58.730 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:58.729+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:59.058 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:27:59.057+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:59.726 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:27:59.725+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:27:59.748 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:27:59.748+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:00.074 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:00.074+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:00.707 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:00.706+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:00.787 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:00.786+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:01.054 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:01.053+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:01.657 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:01.657+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:01.752 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:01.752+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:02.033 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:02.032+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:02.667 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:02.666+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:02.704 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:02.704+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:03.053 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:03.053+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:03.645 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:03.644+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:03.682 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:03.681+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:04.042 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:04.042+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:04.650 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:04.650+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:04.686 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:04.686+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:05.000 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:04.999+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:05.665 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:05.665+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:05.725 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:05.724+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:06.029 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:06.029+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:06.656 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:06.656+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:06.692 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:06.692+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:07.037 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:07.037+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:07.664 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:07.663+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:07.675 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:07.675+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:08.057 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:08.057+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:08.660 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:08.660+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:08.667 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:08.667+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:09.097 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:09.097+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:09.652 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:09.652+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:09.686 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:09.685+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:10.074 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:10.074+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:10.633 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:10.633+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:10.716 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:10.716+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:11.100 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:11.100+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:11.586 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:11.586+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:11.708 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:11.708+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:12.078 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:12.077+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:12.623 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:12.623+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:12.742 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:12.742+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:13.109 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:13.109+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:13.588 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:13.588+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:13.725 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:13.725+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:14.148 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:14.148+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:14.618 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:14.618+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:14.676 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:14.676+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:15.120 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:15.120+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:15.613 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:15.613+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:15.721 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:15.721+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:16.104 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:16.104+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:16.640 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:16.640+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:16.759 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:16.760+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:17.113 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:17.114+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:17.619 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:17.619+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:17.774 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:17.775+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:18.102 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:18.103+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:18.619 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:18.619+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:18.747 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:18.748+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:19.092 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:19.093+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:19.618 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:19.619+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:19.730 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:19.730+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:20.098 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:20.099+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:20.617 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:20.618+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:20.717 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:20.718+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:21.086 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:21.087+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:21.608 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:21.609+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:21.760 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:21.761+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:22.112 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:22.113+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:22.593 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:22.594+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:22.748 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:22.748+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:23.133 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:23.132+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:23.641 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:23.642+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:23.747 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:23.748+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:24.132 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:24.133+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:24.634 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:24.635+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:24.711 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:24.712+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:25.089 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:25.090+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:25.627 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:25.628+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:25.728 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:25.729+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:26.081 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:26.081+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:26.677 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:26.678+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:26.716 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:26.717+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:27.059 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:27.059+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:27.689 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:27.689+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:27.707 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:27.708+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:28.082 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:28.082+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:28.709 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:28.710+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:28.750 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:28.750+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:29.115 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:29.115+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:29.697 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:29.697+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:29.755 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:29.754+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:30.081 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:30.079+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:30.654 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:30.654+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:30.782 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:30.782+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:31.101 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:31.101+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:31.621 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:31.621+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:31.781 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:31.781+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:32.134 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:32.134+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:32.580 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:32.580+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:32.832 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:32.831+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:33.165 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:33.164+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:33.598 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:33.597+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:33.854 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:33.854+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:34.178 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:34.177+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:34.601 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:34.601+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:34.831 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:34.831+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:35.207 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:35.207+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:35.600 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:35.599+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:35.805 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:35.805+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:36.194 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:36.194+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:36.560 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:36.560+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:36.832 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:36.832+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:37.202 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:37.202+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:37.541 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:37.541+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:37.817 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:37.816+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:38.233 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:38.233+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:38.539 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:38.539+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:38.799 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:38.798+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:39.188 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:39.188+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:39.546 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:39.546+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:39.773 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:39.773+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:40.154 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:40.154+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:40.535 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:40.535+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:40.802 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:40.802+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:41.122 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:41.122+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:41.491 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:41.490+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:41.760 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:41.760+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:42.078 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:42.078+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:42.511 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:42.511+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:42.719 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:42.719+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:43.087 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:43.087+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:43.527 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:43.527+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:43.686 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:43.686+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:44.126 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:44.126+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:44.536 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:44.536+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:44.696 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:44.696+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:45.161 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:45.160+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:45.561 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:45.561+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:45.669 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:45.669+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:46.128 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:46.128+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:46.528 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:46.527+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:46.656 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:46.656+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:47.112 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:47.112+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:47.565 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:47.564+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:47.615 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:47.615+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:48.161 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:48.161+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:48.533 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:48.533+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:48.580 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:48.579+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:49.118 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:49.117+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:49.571 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:49.571+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:49.574 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:49.574+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:50.092 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:50.091+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:50.569 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:50.569+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:50.596 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:50.596+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:51.140 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:51.139+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:51.539 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:51.539+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:51.607 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:51.607+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:52.164 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:52.163+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:52.492 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:52.492+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:52.588 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:52.587+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:53.136 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:53.136+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:53.472 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:53.472+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:53.612 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:53.612+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:54.138 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:54.138+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:54.471 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:54.471+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:54.606 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:54.606+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:55.101 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:55.101+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:55.467 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:55.467+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:55.646 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:55.646+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:56.121 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:56.120+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:56.449 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:56.449+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:56.672 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:56.672+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:57.147 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:57.147+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:57.432 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:57.432+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:57.643 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:57.643+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:58.185 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:58.184+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:58.392 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:58.392+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:58.691 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:58.691+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:59.140 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:28:59.140+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:59.377 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:28:59.378+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:28:59.717 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:28:59.717+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:00.173 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:00.172+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:00.405 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:00.405+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:00.751 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:00.751+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:01.137 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:01.138+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:01.433 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:01.433+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:01.766 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:01.767+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:02.115 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:02.116+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:02.434 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:02.435+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:02.739 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:02.740+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:03.148 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:03.149+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:03.427 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:03.427+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:03.694 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:03.695+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:04.169 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:04.170+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:04.382 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:04.383+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:04.689 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:04.690+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:05.127 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:05.128+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:05.403 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:05.404+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:05.682 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:05.682+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:06.082 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:06.083+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:06.444 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:06.445+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:06.658 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:06.659+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:07.117 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:07.117+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:07.402 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:07.403+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:07.693 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:07.693+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:08.071 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:08.072+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:08.415 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:08.415+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:08.720 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:08.721+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:09.043 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:09.043+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:09.398 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:09.398+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:09.746 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:09.746+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:10.074 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:10.074+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:10.430 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:10.431+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:10.732 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:10.732+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:11.088 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:11.089+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:11.429 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:11.429+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:11.692 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:11.692+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:12.075 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:12.076+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:12.429 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:12.430+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:12.732 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:12.733+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:13.123 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:13.123+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:13.450 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:13.450+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:13.706 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:13.705+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:14.083 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:14.083+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:14.486 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:14.485+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:14.739 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:14.739+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:15.036 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:15.035+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:15.450 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:15.450+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:15.764 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:15.763+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:15.990 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:15.989+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:16.457 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:16.457+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:16.743 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:16.743+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:17.023 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:17.023+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:17.440 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:17.440+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:17.743 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:17.742+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:18.061 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:18.060+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:18.412 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:18.412+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:18.769 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:18.769+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:19.073 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:19.072+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:19.394 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:19.394+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:19.810 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:19.809+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:20.115 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:20.115+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:20.390 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:20.390+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:20.761 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:20.761+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:21.102 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:21.102+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:21.391 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:21.391+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:21.725 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:21.725+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:22.113 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:22.112+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:22.387 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:22.387+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:22.749 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:22.749+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:23.135 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:23.135+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:23.435 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:23.435+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:23.769 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:23.769+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:24.124 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:24.124+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:24.474 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:24.473+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:24.793 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:24.793+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:25.097 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:25.097+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:25.451 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:25.451+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:25.827 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:25.827+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:26.097 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:26.097+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:26.424 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:26.423+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:26.878 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:26.877+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:27.095 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:27.094+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:27.427 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:27.426+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:27.894 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:27.893+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:28.099 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:28.099+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:28.415 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:28.415+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:28.845 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:28.845+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:29.057 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:29.056+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:29.412 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:29.412+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:29.809 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:29.809+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:30.031 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:30.031+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:30.450 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:30.450+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:30.785 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:30.785+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:30.992 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:30.992+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:31.436 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:31.436+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:31.833 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:31.833+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:31.951 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:31.950+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:32.432 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:32.432+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:32.851 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:32.851+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:32.973 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:32.973+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:33.412 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:33.412+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:33.817 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:33.816+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:34.016 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:34.015+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:34.439 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:34.438+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:34.791 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:34.791+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:35.045 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:35.045+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:35.451 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:35.450+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:35.801 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:35.801+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:35.996 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:35.996+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:36.401 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:36.401+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:36.804 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:36.804+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:37.019 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:37.019+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:37.384 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:37.383+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:37.835 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:37.835+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:38.045 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:38.045+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:38.404 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:38.404+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:38.813 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:38.812+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:39.037 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:39.037+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:39.434 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:39.434+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:39.818 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:39.817+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:40.055 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:40.055+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:40.465 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:40.464+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:40.841 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:40.841+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:41.055 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:41.055+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:41.507 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:41.506+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:41.892 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:41.891+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:42.080 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:42.079+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:42.459 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:42.460+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:42.878 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:42.878+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:43.047 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:43.047+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:43.509 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:43.508+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:43.859 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:43.859+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:44.027 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:44.026+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:44.543 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:44.542+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:44.906 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:44.906+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:44.997 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:44.998+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:45.583 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:45.584+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:45.885 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:45.885+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:46.045 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:46.045+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:46.628 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:46.628+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:46.912 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:46.913+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:47.006 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:47.007+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:47.640 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:47.641+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:47.912 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:47.913+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:47.964 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:47.964+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:48.620 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:48.621+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:48.865 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:48.866+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:48.959 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:48.960+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:49.583 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:49.584+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:49.871 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:49.872+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:49.942 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:49.943+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:50.575 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:50.576+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:50.868 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:50.869+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:50.959 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:50.959+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:51.596 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:51.597+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:51.867 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:51.868+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:51.966 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:51.967+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:52.602 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:52.602+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:52.907 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:52.907+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:52.924 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:52.925+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:53.600 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:53.601+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:53.914 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:53.915+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:53.945 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:53.946+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:54.606 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:54.607+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:54.943 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:54.944+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:54.956 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:54.957+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:55.599 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:55.600+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:55.953 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:55.954+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:56.000 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:56.001+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:56.594 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:56.595+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:56.968 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:56.968+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:56.981 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:56.982+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:57.563 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:57.564+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:57.944 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:57.944+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:57.957 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:57.957+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:58.612 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:58.612+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:58.974 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:58.974+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:58.996 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:29:58.996+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:59.565 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:29:59.565+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:29:59.977 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:29:59.977+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:00.045 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:00.043+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:00.543 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:00.543+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:01.003 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:01.003+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:01.085 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:01.085+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:01.519 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:01.519+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:02.020 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:02.020+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:02.089 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:02.089+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:02.477 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:02.477+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:02.980 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:02.980+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:03.105 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:03.105+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:03.476 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:03.476+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:04.030 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:04.030+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:04.136 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:04.136+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:04.487 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:04.487+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:05.011 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:05.011+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:05.134 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:05.134+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:05.532 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:05.532+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:05.966 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:05.966+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:06.120 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:06.119+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:06.490 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:06.490+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:07.001 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:07.001+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:07.134 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:07.134+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:07.459 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:07.459+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:07.960 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:07.959+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:08.086 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:08.085+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:08.472 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:08.471+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:09.002 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:09.002+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:09.053 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:09.053+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:09.490 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:09.490+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:10.014 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:10.014+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:10.104 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:10.103+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:10.513 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:10.513+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:10.996 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:10.996+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:11.072 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:11.071+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:11.501 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:11.500+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:12.043 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:12.042+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:12.076 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:12.076+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:12.533 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:12.533+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:13.019 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:13.019+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:13.093 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:13.093+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:13.507 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:13.507+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:14.002 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:14.002+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:14.060 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:14.060+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:14.516 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:14.515+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:14.993 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:14.993+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:15.082 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:15.082+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:15.549 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:15.549+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:15.962 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:15.961+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:16.092 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:16.092+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:16.566 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:16.566+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:17.006 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:17.006+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:17.050 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:17.050+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:17.528 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:17.528+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:18.011 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:18.010+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:18.032 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:18.032+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:18.510 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:18.510+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:19.006 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:19.006+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:19.008 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:19.008+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:19.535 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:19.535+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:19.994 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:19.994+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:20.026 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:20.026+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:20.513 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:20.513+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:20.967 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:20.967+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:20.994 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:20.994+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:21.527 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:21.526+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:21.929 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:21.928+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:21.953 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:21.951+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:22.518 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:22.517+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:22.911 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:22.910+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:22.918 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:22.918+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:23.555 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:23.554+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:23.908 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:23.908+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:23.920 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:23.920+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:24.551 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:24.551+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:24.877 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:24.877+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:24.946 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:24.946+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:25.535 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:25.535+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:25.868 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:25.867+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:25.924 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:25.924+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:26.566 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:26.566+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:26.877 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:26.877+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:26.932 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:26.932+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:27.609 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:27.609+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:27.857 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:27.857+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:27.982 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:27.983+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:28.639 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:28.639+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:28.858 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:28.858+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:28.965 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:28.966+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:29.593 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:29.594+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:29.821 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:29.821+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:29.998 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:29.999+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:30.624 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:30.625+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:30.783 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:30.784+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:30.972 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:30.973+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:31.654 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:31.655+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:31.777 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:31.778+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:31.941 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:31.942+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:32.673 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:32.673+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:32.778 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:32.778+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:32.912 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:32.912+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:33.670 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:33.671+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:33.796 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:33.796+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:33.889 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:33.890+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:34.633 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:34.634+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:34.753 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:34.754+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:34.841 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:34.842+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:35.650 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:35.651+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:35.748 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:35.749+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:35.846 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:35.846+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:36.699 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:36.699+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:36.748 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:36.749+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:36.798 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:36.799+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:37.721 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:37.721+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:37.726 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:37.727+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:37.817 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:37.818+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:38.733 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:38.734+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:38.736 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:38.737+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:38.782 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:38.783+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:39.693 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:39.694+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:39.704 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:39.705+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:39.772 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:39.773+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:40.676 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:40.677+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:40.679 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:40.680+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:40.809 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:40.810+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:41.701 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:41.702+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:41.713 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:41.714+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:41.780 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:41.781+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:42.679 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:42.678+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:42.730 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:42.730+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:42.771 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:42.770+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:43.659 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:43.658+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:43.733 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:43.732+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:43.737 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:43.737+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:44.613 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:44.613+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:44.708 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:44.707+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:44.773 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:44.773+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:45.658 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:45.657+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:45.689 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:45.689+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:45.777 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:45.776+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:46.617 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:46.617+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:46.713 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:46.713+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:46.754 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:46.753+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:47.624 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:47.623+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:47.760 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:47.760+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:47.771 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:47.771+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:48.575 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:48.575+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:48.781 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:48.781+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:48.789 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:48.789+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:49.565 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:49.565+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:49.771 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:49.771+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:49.796 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:49.796+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:50.559 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:50.558+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:50.783 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:50.783+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:50.800 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:50.800+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:51.574 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:51.574+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:51.754 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:51.754+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:51.805 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:51.805+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:52.551 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:52.551+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:52.722 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:52.722+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:52.814 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:52.813+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:53.539 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:53.538+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:53.731 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:53.731+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:53.837 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:53.837+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:54.589 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:54.589+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:54.723 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:54.722+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:54.793 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:54.793+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:55.559 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:55.558+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:55.747 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:55.747+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:55.803 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:55.803+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:56.547 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:56.546+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:56.760 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:56.760+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:56.837 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:56.837+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:57.504 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:57.504+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:57.803 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:57.802+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:57.858 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:57.858+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:58.494 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:58.494+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:58.846 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:58.846+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:58.852 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:58.851+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:59.457 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:30:59.456+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:59.869 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:30:59.869+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:30:59.884 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:30:59.884+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:00.463 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:00.462+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:00.839 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:00.839+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:00.874 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:00.874+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:01.504 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:01.503+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:01.853 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:01.852+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:01.853 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:01.853+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:02.500 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:02.500+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:02.831 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:02.831+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:02.844 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:02.844+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:03.509 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:03.508+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:03.827 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:03.827+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:03.868 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:03.867+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:04.538 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:04.538+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:04.826 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:04.826+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:04.834 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:04.834+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:05.581 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:05.580+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:05.803 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:05.802+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:05.833 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:05.832+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:06.629 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:06.629+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:06.818 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:06.817+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:06.868 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:06.868+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:07.663 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:07.663+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:07.831 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:07.830+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:07.867 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:07.866+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:08.672 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:08.672+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:08.805 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:08.805+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:08.887 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:08.886+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:09.693 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:09.693+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:09.770 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:09.769+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:09.896 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:09.895+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:10.708 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:10.708+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:10.754 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:10.754+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:10.906 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:10.906+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:11.739 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:11.739+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:11.781 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:11.781+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:11.873 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:11.871+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:12.716 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:12.716+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:12.808 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:12.808+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:12.874 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:12.874+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:13.757 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:13.758+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:13.850 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:13.850+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:13.903 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:13.904+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:14.712 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:14.713+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:14.870 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:14.870+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:14.925 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:14.926+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:15.732 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:15.733+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:15.903 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:15.904+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:15.947 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:15.948+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:16.742 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:16.743+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:16.857 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:16.858+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:16.962 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:16.963+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:17.732 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:17.732+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:17.832 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:17.832+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:17.925 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:17.925+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:18.696 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:18.696+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:18.858 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:18.859+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:18.949 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:18.949+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:19.652 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:19.652+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:19.828 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:19.829+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:19.968 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:19.969+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:20.663 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:20.664+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:20.795 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:20.795+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:20.944 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:20.945+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:21.687 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:21.687+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:21.765 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:21.766+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:21.902 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:21.902+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:22.712 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:22.712+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:22.770 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:22.771+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:22.909 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:22.910+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:23.690 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:23.691+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:23.745 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:23.745+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:23.928 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:23.929+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:24.733 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:24.730+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:24.755 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:24.756+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:24.902 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:24.903+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:25.766 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:25.766+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:25.771 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:25.771+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:25.874 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:25.874+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:26.722 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:26.721+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:26.807 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:26.807+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:26.826 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:26.826+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:27.765 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:27.764+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:27.807 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:27.807+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:27.873 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:27.873+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:28.759 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:28.759+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:28.778 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:28.777+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:28.905 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:28.905+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:29.712 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:29.711+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:29.823 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:29.823+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:29.896 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:29.896+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:30.748 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:30.748+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:30.856 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:30.856+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:30.859 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:30.859+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:31.746 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:31.745+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:31.824 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:31.823+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:31.890 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:31.890+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:32.752 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:32.751+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:32.826 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:32.826+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:32.901 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:32.901+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:33.770 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:33.770+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:33.814 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:33.814+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:33.896 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:33.896+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:34.781 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:34.780+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:34.829 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:34.828+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:34.918 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:34.918+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:35.753 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:35.752+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:35.780 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:35.780+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:35.963 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:35.963+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:36.754 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:36.753+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:36.778 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:36.777+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:36.964 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:36.964+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:37.744 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:37.744+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:37.798 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:37.798+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:37.998 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:37.998+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:38.774 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:38.774+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:38.836 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:38.836+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:38.969 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:38.968+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:39.733 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:39.732+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:39.793 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:39.792+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:39.976 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:39.975+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:40.745 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:40.745+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:40.800 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:40.800+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:41.012 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:41.011+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:41.760 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:41.760+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:41.844 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:41.844+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:41.975 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:41.975+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:42.719 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:42.719+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:42.834 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:42.834+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:42.992 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:42.992+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:43.692 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:43.692+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:43.853 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:43.852+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:43.974 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:43.974+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:44.645 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:44.644+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:44.881 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:44.881+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:45.010 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:45.009+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:45.641 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:45.641+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:45.864 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:45.864+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:46.054 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:46.054+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:46.663 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:46.663+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:46.826 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:46.825+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:47.047 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:47.047+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:47.631 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:47.630+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:47.816 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:47.816+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:48.071 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:48.070+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:48.622 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:48.622+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:48.807 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:48.807+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:49.072 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:49.071+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:49.608 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:49.608+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:49.800 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:49.800+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:50.120 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:50.120+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:50.605 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:50.605+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:50.795 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:50.795+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:51.161 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:51.159+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:51.653 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:51.653+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:51.763 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:51.763+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:52.209 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:52.209+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:52.684 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:52.684+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:52.806 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:52.806+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:53.237 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:53.236+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:53.696 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:53.696+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:53.844 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:53.844+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:54.245 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:54.244+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:54.718 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:54.719+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:54.851 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:54.850+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:55.253 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:55.253+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:55.750 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:55.750+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:55.901 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:55.901+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:56.299 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:56.298+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:56.759 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:56.760+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:56.914 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:56.915+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:57.345 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:57.346+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:57.785 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:57.786+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:57.961 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:57.961+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:58.310 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:58.311+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:58.824 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:58.824+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:59.003 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:59.004+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:59.266 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:31:59.267+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:59.814 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:31:59.815+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:31:59.985 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:31:59.986+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:00.265 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:00.265+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:00.783 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:00.784+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:01.030 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:01.031+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:01.234 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:01.234+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:01.795 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:01.796+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:02.034 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:02.035+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:02.248 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:02.249+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:02.786 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:02.786+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:03.068 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:03.069+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:03.298 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:03.299+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:03.754 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:03.754+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:04.082 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:04.083+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:04.283 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:04.284+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:04.735 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:04.736+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:05.106 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:05.106+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:05.262 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:05.262+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:05.737 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:05.737+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:06.064 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:06.065+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:06.276 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:06.277+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:06.732 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:06.733+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:07.075 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:07.076+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:07.271 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:07.272+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:07.741 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:07.741+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:08.052 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:08.053+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:08.239 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:08.240+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:08.725 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:08.725+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:09.010 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:09.010+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:09.206 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:09.206+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:09.739 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:09.739+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:10.026 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:10.026+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:10.210 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:10.210+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:10.731 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:10.731+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:11.070 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:11.070+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:11.257 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:11.257+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:11.721 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:11.720+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:12.092 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:12.092+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:12.230 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:12.229+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:12.759 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:12.758+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:13.109 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:13.109+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:13.258 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:13.258+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:13.787 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:13.787+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:14.084 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:14.084+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:14.294 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:14.290+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:14.807 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:14.806+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:15.061 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:15.061+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:15.278 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:15.277+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:15.823 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:15.823+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:16.021 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:16.021+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:16.281 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:16.281+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:16.856 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:16.856+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:17.012 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:17.012+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:17.309 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:17.308+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:17.843 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:17.843+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:18.041 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:18.040+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:18.353 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:18.353+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:18.794 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:18.793+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:19.061 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:19.061+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:19.376 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:19.375+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:19.794 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:19.794+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:20.086 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:20.086+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:20.369 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:20.368+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:20.823 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:20.823+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:21.051 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:21.050+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:21.385 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:21.384+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:21.855 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:21.855+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:22.018 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:22.018+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:22.398 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:22.398+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:22.846 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:22.846+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:23.040 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:23.040+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:23.356 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:23.356+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:23.874 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:23.873+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:24.086 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:24.086+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:24.336 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:24.335+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:24.893 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:24.892+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:25.134 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:25.134+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:25.299 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:25.299+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:25.853 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:25.853+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:26.130 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:26.130+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:26.295 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:26.295+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:26.897 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:26.897+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:27.100 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:27.100+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:27.341 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:27.340+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:27.912 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:27.911+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:28.123 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:28.122+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:28.328 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:28.328+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:28.923 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:28.923+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:29.123 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:29.122+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:29.341 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:29.341+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:29.880 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:29.879+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:30.091 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:30.090+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:30.350 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:30.350+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:30.909 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:30.909+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:31.118 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:31.118+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:31.394 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:31.394+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:31.928 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:31.927+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:32.110 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:32.110+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:32.348 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:32.348+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:32.967 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:32.966+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:33.112 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:33.111+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:33.326 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:33.325+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:33.973 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:33.972+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:34.091 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:34.091+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:34.376 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:34.375+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:34.971 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:34.970+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:35.111 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:35.111+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:35.355 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:35.355+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:35.976 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:35.976+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:36.076 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:36.076+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:36.392 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:36.392+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:36.952 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:36.952+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:37.031 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:37.031+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:37.405 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:37.405+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:37.915 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:37.915+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:38.025 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:38.024+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:38.367 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:38.366+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:38.944 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:38.945+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:39.043 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:39.043+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:39.369 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:39.369+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:39.938 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:39.938+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:40.071 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:40.070+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:40.363 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:40.364+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:40.951 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:40.951+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 10 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:41.081 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:41.081+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:41.352 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:41.353+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:41.941 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:41.941+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 9 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:42.063 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:42.064+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:42.322 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:42.323+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:42.893 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:42.894+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 9 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:43.113 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:43.114+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:43.329 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:43.330+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:43.848 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:43.849+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 9 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:44.160 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:44.161+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:44.365 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:44.366+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:44.829 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:44.829+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 9 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:45.133 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:45.134+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:45.344 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:45.344+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:45.878 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:45.879+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 9 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:46.142 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:46.143+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:46.332 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:46.333+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:46.903 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:46.903+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 9 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:47.107 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:47.108+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:47.348 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:47.348+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:47.943 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:47.944+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 9 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:48.080 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:48.080+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:48.392 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:48.393+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:48.925 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:48.925+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 9 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:49.039 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:49.040+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:49.348 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:49.349+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:49.953 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:49.954+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 9 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:50.011 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:50.012+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:50.326 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:50.326+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:50.906 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:50.907+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 9 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:51.001 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:51.001+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:51.307 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:51.308+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:51.942 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:51.943+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 9 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:52.036 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:52.037+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:52.287 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:52.288+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:52.927 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:52.927+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 9 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:53.076 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:53.076+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:53.323 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:53.323+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:53.919 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:53.919+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 9 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:54.074 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:54.073+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:54.283 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:54.283+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:54.869 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:54.869+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 9 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:55.094 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:55.094+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:55.310 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:55.310+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:55.889 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:55.889+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 9 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:56.058 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:56.057+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:56.311 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:56.311+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:56.917 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:56.916+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 9 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:57.015 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:57.015+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:57.292 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:57.292+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:57.892 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:57.891+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 9 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:58.035 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:58.035+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:58.318 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:58.317+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:58.873 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:58.872+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 9 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:59.029 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:32:59.029+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:59.332 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:32:59.332+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:32:59.880 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:32:59.880+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 9 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:00.010 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:00.009+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:00.301 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:00.301+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:00.868 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:00.868+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 9 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:01.047 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:01.046+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:01.313 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:01.312+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:01.888 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:01.887+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 9 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:02.019 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:02.018+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:02.318 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:02.318+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:02.872 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:02.872+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 9 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:03.029 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:03.029+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:03.326 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:03.326+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:03.824 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:03.823+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 9 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:04.036 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:04.036+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:04.340 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:04.339+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:04.836 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:04.836+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 9 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:05.004 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:05.004+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:05.367 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:05.366+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:05.868 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:05.868+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 9 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:05.992 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:05.991+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:06.397 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:06.396+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:06.839 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:06.839+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 9 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:07.010 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:07.010+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:07.361 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:07.360+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:07.850 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:07.849+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 9 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:07.989 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:07.989+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:08.324 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:08.324+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:08.839 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:08.838+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 9 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:08.995 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:08.995+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:09.308 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:09.307+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:09.853 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:09.853+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 9 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:09.966 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:09.966+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:10.321 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:10.320+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:10.843 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:10.842+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 9 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:10.959 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:10.959+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:11.316 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:11.315+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:11.886 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:11.886+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:11.934 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:11.934+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:12.306 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:12.306+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:12.885 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:12.885+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:12.978 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:12.978+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:13.339 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:13.339+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:13.895 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:13.895+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:13.960 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:13.959+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:14.365 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:14.364+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:14.864 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:14.864+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:14.994 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:14.994+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:15.400 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:15.399+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:15.897 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:15.897+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:15.960 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:15.960+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:16.374 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:16.373+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:16.937 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:16.936+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:16.953 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:16.953+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:17.420 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:17.420+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:17.917 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:17.916+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:17.979 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:17.978+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:18.383 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:18.382+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:18.935 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:18.934+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:18.956 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:18.955+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:19.365 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:19.365+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:19.944 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:19.944+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:20.004 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:20.004+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:20.385 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:20.385+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:20.910 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:20.910+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:20.969 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:20.969+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:21.412 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:21.411+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:21.920 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:21.919+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:21.987 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:21.986+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:22.395 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:22.395+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:22.906 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:22.905+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:23.027 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:23.027+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:23.359 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:23.358+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:23.936 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:23.936+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:23.991 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:23.991+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:24.334 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:24.333+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:24.898 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:24.899+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:24.955 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:24.956+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:25.307 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:25.308+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:25.926 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:25.927+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:26.000 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:26.001+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:26.288 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:26.289+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:26.949 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:26.950+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:27.036 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:27.037+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:27.265 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:27.265+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:27.986 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:27.987+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:28.000 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:28.001+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:28.302 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:28.303+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:28.949 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:28.949+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:28.978 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:28.979+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:29.335 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:29.336+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:29.968 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:29.969+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:30.003 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:30.003+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:30.361 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:30.361+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:30.984 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:30.985+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:31.038 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:31.038+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:31.350 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:31.351+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:32.015 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:32.016+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:32.063 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:32.064+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:32.305 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:32.306+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:33.013 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:33.014+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:33.061 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:33.062+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:33.294 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:33.294+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:33.970 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:33.971+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:34.048 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:34.049+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:34.267 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:34.268+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:34.990 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:34.990+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:35.023 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:35.024+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:35.272 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:35.273+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:35.994 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:35.995+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:36.027 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:36.027+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:36.291 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:36.291+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:37.013 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:37.014+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:37.039 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:37.040+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:37.320 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:37.321+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:37.995 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:37.995+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:38.020 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:38.020+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:38.305 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:38.304+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:39.016 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:39.016+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:39.023 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:39.022+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:39.303 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:39.302+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:39.986 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:39.986+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:40.003 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:40.003+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:40.335 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:40.335+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:40.977 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:40.976+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:41.026 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:41.026+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:41.336 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:41.335+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:41.952 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:41.952+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:41.986 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:41.985+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:42.368 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:42.368+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:42.943 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:42.943+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:43.017 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:43.016+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:43.391 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:43.391+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:43.951 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:43.950+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:44.020 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:44.020+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:44.424 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:44.423+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:44.951 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:44.951+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:45.056 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:45.056+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:45.435 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:45.435+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:45.927 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:45.927+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:46.018 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:46.018+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:46.464 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:46.463+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:46.923 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:46.923+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:46.971 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:46.971+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:47.435 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:47.435+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:47.890 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:47.889+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:47.955 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:47.955+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:48.395 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:48.395+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:48.921 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:48.920+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:48.927 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:48.926+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:49.365 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:49.365+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:49.939 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:49.939+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:49.966 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:49.965+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:50.318 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:50.317+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:50.924 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:50.923+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:50.956 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:50.956+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:51.349 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:51.348+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:51.927 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:51.926+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:51.927 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:51.926+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:52.322 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:52.322+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:52.885 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:52.885+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:52.944 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:52.944+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:53.295 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:53.295+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:53.897 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:53.896+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:53.954 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:53.953+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:54.287 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:54.287+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:54.909 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:54.909+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:54.935 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:54.935+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:55.261 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:55.261+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:55.926 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:55.926+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:55.944 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:55.944+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:56.266 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:56.266+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:56.888 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:56.888+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:56.970 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:56.970+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:57.220 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:57.220+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:57.878 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:57.877+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:58.002 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:58.002+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:58.257 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:58.257+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:58.898 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:58.898+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:58.971 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:58.970+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:59.268 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:33:59.267+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:59.865 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:33:59.864+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:33:59.961 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:33:59.961+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:00.290 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:00.289+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:00.849 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:00.849+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:01.004 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:01.004+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:01.297 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:01.296+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:01.841 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:01.841+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:01.999 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:01.999+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:02.339 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:02.339+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:02.825 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:02.824+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:03.015 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:03.014+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:03.350 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:03.349+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:03.860 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:03.859+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:04.064 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:04.063+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:04.302 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:04.302+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:04.892 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:04.892+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:05.065 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:05.064+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:05.263 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:05.262+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:05.895 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:05.895+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:06.043 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:06.042+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:06.263 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:06.262+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:06.880 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:06.879+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:07.028 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:07.028+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:07.285 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:07.285+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:07.862 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:07.861+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:08.024 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:08.023+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:08.297 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:08.296+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:08.907 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:08.908+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:09.001 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:09.001+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:09.262 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:09.262+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:09.889 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:09.889+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:10.015 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:10.016+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:10.311 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:10.311+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:10.929 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:10.929+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:11.009 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:11.007+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:11.340 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:11.340+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:11.952 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:11.952+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:12.018 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:12.018+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:12.372 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:12.372+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:12.971 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:12.972+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:13.018 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:13.019+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:13.396 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:13.396+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:13.959 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:13.960+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:14.058 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:14.058+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:14.358 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:14.358+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:14.937 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:14.938+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:15.094 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:15.095+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:15.365 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:15.365+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:15.934 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:15.934+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:16.145 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:16.146+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:16.386 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:16.387+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:16.908 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:16.908+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:17.150 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:17.150+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:17.430 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:17.431+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:17.953 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:17.953+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:18.142 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:18.142+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:18.479 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:18.479+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:18.991 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:18.991+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:19.138 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:19.139+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:19.472 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:19.473+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:19.955 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:19.955+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:20.109 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:20.110+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:20.424 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:20.425+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:20.966 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:20.967+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:21.105 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:21.106+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:21.449 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:21.449+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:21.991 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:21.991+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:22.081 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:22.081+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:22.462 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:22.461+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:23.038 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:23.037+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:23.078 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:23.078+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:23.488 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:23.488+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:24.033 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:24.033+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:24.064 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:24.064+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:24.474 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:24.474+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:25.059 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:25.059+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:25.097 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:25.097+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:25.474 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:25.474+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:26.053 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:26.052+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:26.065 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:26.064+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:26.458 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:26.458+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:27.032 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:27.032+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:27.085 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:27.085+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:27.416 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:27.416+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:28.034 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:28.034+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:28.100 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:28.100+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:28.400 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:28.400+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:29.055 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:29.055+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:29.075 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:29.075+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:29.398 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:29.398+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:30.081 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:30.081+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:30.126 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:30.125+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:30.406 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:30.405+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:31.097 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:31.097+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:31.124 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:31.124+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:31.416 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:31.416+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:32.125 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:32.124+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:32.166 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:32.166+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:32.384 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:32.383+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:33.111 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:33.111+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:33.141 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:33.139+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:33.356 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:33.356+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:34.131 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:34.131+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:34.149 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:34.148+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:34.352 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:34.352+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:35.121 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:35.121+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:35.135 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:35.135+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:35.372 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:35.371+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:36.088 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:36.088+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:36.136 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:36.136+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:36.397 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:36.396+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:37.042 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:37.041+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:37.088 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:37.087+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:37.360 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:37.360+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:38.057 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:38.057+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:38.091 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:38.091+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:38.334 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:38.334+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:39.054 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:39.053+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:39.061 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:39.061+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:39.376 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:39.376+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:40.061 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:40.060+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:40.089 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:40.089+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:40.366 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:40.366+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:41.082 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:41.082+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:41.122 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:41.121+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:41.319 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:41.319+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:42.059 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:42.059+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:42.126 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:42.125+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:42.359 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:42.359+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:43.086 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:43.086+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:43.090 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:43.090+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:43.395 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:43.395+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:44.066 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:44.066+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 12 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:44.130 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:44.130+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:44.358 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:44.358+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:45.077 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:45.077+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 11 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:45.157 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:45.157+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:45.346 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:45.346+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:46.109 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:46.109+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 11 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:46.156 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:46.155+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:46.322 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:46.322+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:47.093 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:47.092+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 11 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:47.121 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:47.121+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:47.370 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:47.369+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:48.123 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:48.122+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 11 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:48.152 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:48.152+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:48.378 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:48.377+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:49.140 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:49.138+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:49.152 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:49.152+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 11 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:49.349 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:49.349+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:50.154 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:50.153+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:50.158 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:50.159+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 11 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:50.334 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:50.334+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:51.139 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:51.140+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 11 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:51.156 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:51.156+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:51.313 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:51.312+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:52.093 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:52.093+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 11 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:52.200 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:52.200+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:52.284 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:52.284+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:53.108 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:53.109+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 11 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:53.178 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:53.179+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:53.254 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:53.255+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:54.155 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:54.156+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 11 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:54.159 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:54.160+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:54.219 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:54.219+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:55.158 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:55.159+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:55.188 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:55.189+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 11 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:55.255 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:55.255+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:56.113 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:56.113+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:56.165 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:56.166+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 11 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:56.288 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:56.288+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:57.141 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:57.141+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:57.148 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:57.148+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 11 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:57.325 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:57.325+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:58.169 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:58.169+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:58.182 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:58.183+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 11 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:58.304 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:58.305+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:59.160 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:34:59.161+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:59.225 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:34:59.225+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 11 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:34:59.345 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:34:59.346+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:00.145 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:00.146+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:00.227 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:00.227+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 11 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:00.330 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:00.331+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:01.193 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:01.194+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:01.208 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:01.208+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 11 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:01.293 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:01.294+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:02.177 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:02.178+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 11 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:02.189 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:02.189+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:02.336 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:02.337+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:03.145 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:03.146+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 11 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:03.215 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:03.215+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:03.339 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:03.339+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:04.150 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:04.150+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 11 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:04.220 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:04.218+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:04.298 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:04.299+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:05.187 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:05.188+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 11 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:05.204 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:05.204+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:05.293 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:05.292+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:06.170 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:06.170+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:06.203 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:06.203+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 11 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:06.317 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:06.317+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:07.175 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:07.175+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:07.224 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:07.223+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 11 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:07.311 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:07.311+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:08.188 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:08.188+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:08.268 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:08.267+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 11 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:08.320 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:08.320+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:09.164 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:09.164+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:09.309 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:09.308+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 11 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:09.328 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:09.328+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:10.143 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:10.143+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:10.299 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:10.299+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 11 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:10.355 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:10.354+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:11.172 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:11.172+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:11.333 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:11.333+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 11 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:11.362 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:11.362+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:12.168 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:12.168+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:12.343 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:12.343+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 11 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:12.384 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:12.384+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:13.152 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:13.152+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:13.338 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:13.338+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 11 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:13.396 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:13.396+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:14.111 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:14.110+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:14.377 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:14.377+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 11 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:14.439 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:14.439+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:15.097 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:15.096+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:15.380 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:15.379+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:15.431 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:15.430+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:16.054 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:16.054+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:16.390 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:16.389+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:16.415 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:16.414+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:17.023 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:17.022+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:17.397 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:17.397+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:17.442 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:17.441+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:18.058 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:18.058+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:18.349 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:18.348+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:18.477 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:18.477+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:19.031 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:19.031+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:19.313 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:19.313+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:19.429 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:19.428+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:20.077 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:20.077+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:20.316 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:20.315+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:20.417 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:20.416+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:21.077 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:21.076+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:21.346 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:21.346+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:21.457 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:21.457+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:22.105 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:22.104+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:22.378 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:22.378+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:22.414 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:22.414+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:23.071 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:23.070+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:23.397 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:23.397+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:23.463 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:23.463+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:24.030 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:24.030+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:24.362 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:24.362+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:24.459 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:24.459+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:25.050 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:25.050+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:25.385 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:25.385+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:25.467 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:25.467+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:26.056 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:26.055+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:26.352 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:26.352+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:26.450 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:26.450+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:27.049 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:27.048+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:27.335 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:27.334+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:27.483 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:27.483+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:28.008 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:28.008+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:28.360 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:28.360+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:28.530 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:28.530+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:29.033 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:29.032+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:29.406 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:29.406+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:29.544 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:29.544+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:30.062 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:30.061+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:30.438 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:30.437+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:30.495 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:30.495+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:31.082 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:31.082+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:31.473 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:31.472+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:31.479 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:31.479+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:32.036 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:32.035+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:32.429 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:32.428+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:32.486 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:32.485+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:32.987 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:32.987+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:33.462 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:33.462+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:33.507 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:33.506+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:33.971 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:33.970+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:34.496 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:34.496+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:34.530 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:34.530+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:34.974 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:34.974+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:35.535 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:35.535+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:35.564 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:35.563+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:36.020 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:36.020+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:36.542 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:36.542+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:36.584 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:36.585+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:37.055 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:37.056+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:37.557 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:37.557+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:37.590 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:37.591+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:38.009 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:38.010+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:38.582 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:38.582+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:38.600 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:38.600+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:39.004 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:39.004+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:39.595 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:39.595+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:39.619 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:39.620+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:39.965 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:39.966+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:40.575 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:40.575+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:40.659 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:40.660+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:41.006 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:41.007+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:41.578 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:41.578+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:41.687 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:41.688+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:42.013 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:42.014+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:42.570 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:42.570+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:42.669 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:42.669+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:42.967 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:42.968+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:43.588 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:43.589+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:43.682 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:43.682+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:43.951 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:43.952+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:44.548 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:44.548+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:44.730 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:44.731+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:44.946 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:44.947+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:45.510 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:45.511+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:45.744 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:45.745+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:45.961 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:45.962+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:46.484 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:46.484+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:46.747 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:46.748+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:46.917 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:46.917+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:47.507 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:47.508+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:47.737 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:47.737+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:47.954 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:47.954+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:48.547 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:48.547+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:48.705 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:48.704+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:48.936 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:48.935+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:49.579 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:49.579+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:49.746 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:49.745+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:49.891 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:49.891+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:50.543 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:50.542+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:50.760 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:50.760+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:50.848 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:50.847+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:51.561 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:51.561+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:51.734 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:51.734+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:51.896 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:51.895+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:52.538 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:52.537+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:52.759 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:52.759+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:52.857 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:52.857+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:53.564 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:53.563+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:53.715 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:53.714+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:53.873 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:53.873+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:54.538 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:54.538+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:54.709 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:54.708+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:54.831 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:54.830+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:55.540 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:55.539+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:55.677 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:55.677+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:55.834 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:55.834+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:56.533 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:56.533+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:56.693 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:56.693+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:56.803 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:56.803+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:57.554 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:57.554+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:57.704 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:57.703+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:57.829 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:57.829+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:58.575 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:58.575+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:58.707 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:58.707+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:58.805 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:58.805+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:59.609 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:35:59.608+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:59.705 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:35:59.705+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:35:59.776 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:35:59.775+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:00.603 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:00.602+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:00.661 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:00.661+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:00.820 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:00.820+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:01.621 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:01.621+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:01.673 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:01.672+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:01.786 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:01.785+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:02.644 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:02.643+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:02.692 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:02.691+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:02.769 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:02.769+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:03.609 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:03.608+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:03.694 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:03.694+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:03.737 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:03.735+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:04.577 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:04.576+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:04.712 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:04.712+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:04.732 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:04.732+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:05.559 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:05.558+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:05.748 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:05.748+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:05.766 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:05.765+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:06.523 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:06.522+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:06.775 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:06.775+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:06.792 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:06.792+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:07.554 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:07.554+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:07.778 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:07.777+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:07.822 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:07.821+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:08.569 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:08.569+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:08.750 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:08.749+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:08.854 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:08.854+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:09.578 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:09.578+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:09.733 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:09.732+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:09.829 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:09.829+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:10.554 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:10.554+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:10.725 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:10.725+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:10.871 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:10.871+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:11.546 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:11.546+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:11.722 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:11.722+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:11.915 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:11.915+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:12.578 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:12.578+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:12.713 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:12.713+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:12.927 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:12.927+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:13.616 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:13.616+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:13.712 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:13.712+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:13.887 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:13.887+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:14.583 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:14.583+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:14.678 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:14.679+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:14.895 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:14.894+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:15.564 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:15.564+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:15.704 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:15.703+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:15.861 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:15.861+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:16.534 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:16.534+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:16.679 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:16.679+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:16.889 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:16.889+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:17.547 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:17.546+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:17.643 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:17.643+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:17.889 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:17.888+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:18.502 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:18.501+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:18.674 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:18.674+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:18.913 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:18.914+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:19.494 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:19.494+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:19.717 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:19.717+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:19.936 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:19.936+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:20.469 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:20.470+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:20.756 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:20.757+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:20.924 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:20.924+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:21.424 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:21.425+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:21.777 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:21.778+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:21.916 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:21.916+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:22.376 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:22.376+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:22.750 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:22.750+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:22.928 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:22.928+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:23.413 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:23.413+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:23.715 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:23.715+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:23.950 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:23.950+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:24.455 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:24.455+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:24.698 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:24.699+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:24.954 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:24.955+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:25.483 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:25.484+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:25.682 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:25.683+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:26.000 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:26.000+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:26.489 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:26.489+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:26.719 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:26.719+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:27.032 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:27.032+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:27.486 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:27.487+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:27.763 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:27.763+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:27.993 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:27.994+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:28.481 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:28.481+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:28.770 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:28.770+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:29.042 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:29.043+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:29.523 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:29.524+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:29.756 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:29.756+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:29.995 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:29.995+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:30.524 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:30.524+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:30.765 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:30.765+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:31.007 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:31.008+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:31.537 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:31.536+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:31.791 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:31.790+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:31.966 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:31.965+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:32.573 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:32.573+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:32.830 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:32.829+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:33.014 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:33.013+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:33.537 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:33.536+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:33.805 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:33.805+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:33.971 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:33.971+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:34.557 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:34.557+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:34.780 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:34.779+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:34.960 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:34.960+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:35.525 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:35.524+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:35.804 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:35.804+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:35.934 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:35.934+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:36.527 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:36.527+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:36.817 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:36.816+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:36.949 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:36.948+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:37.523 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:37.523+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:37.844 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:37.843+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:37.965 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:37.965+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:38.556 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:38.556+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:38.885 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:38.884+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:38.977 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:38.977+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:39.518 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:39.518+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:39.868 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:39.868+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:39.997 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:39.996+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:40.469 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:40.469+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:40.910 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:40.909+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:40.995 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:40.995+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:41.480 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:41.480+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:41.958 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:41.958+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:41.979 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:41.978+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:42.491 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:42.491+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:42.980 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:42.979+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:43.020 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:43.019+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:43.482 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:43.481+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:43.956 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:43.955+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:44.001 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:44.000+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:44.465 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:44.464+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:44.926 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:44.925+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:44.955 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:44.954+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:45.505 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:45.505+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:45.965 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:45.965+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:45.968 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:45.968+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:46.536 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:46.536+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:46.971 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:46.971+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:46.985 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:46.984+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:47.512 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:47.511+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:47.943 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:47.942+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:48.003 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:48.003+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:48.463 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:48.463+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:48.979 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:48.979+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:49.036 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:49.036+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:49.470 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:49.469+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:49.959 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:49.958+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:50.038 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:50.038+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:50.456 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:50.455+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:50.985 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:50.984+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:51.085 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:51.085+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:51.469 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:51.469+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:52.016 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:52.015+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:52.084 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:52.084+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:52.463 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:52.462+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:52.998 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:52.997+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:53.090 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:53.089+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:53.504 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:53.504+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:54.009 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:54.005+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:54.124 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:54.123+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:54.499 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:54.498+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:54.991 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:54.990+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:55.150 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:55.149+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:55.521 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:55.521+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:55.975 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:55.975+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:56.123 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:56.123+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:56.546 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:56.545+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:56.930 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:56.929+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:57.131 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:57.130+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:57.549 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:57.549+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:57.913 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:57.913+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:58.126 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:58.126+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:58.519 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:58.519+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:58.917 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:58.916+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:59.100 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:36:59.100+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:59.476 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:36:59.476+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:36:59.953 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:36:59.953+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:00.101 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:00.101+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:00.448 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:00.448+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:00.911 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:00.910+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:01.098 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:01.097+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:01.483 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:01.483+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:01.919 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:01.919+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:02.113 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:02.113+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:02.483 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:02.483+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:02.933 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:02.934+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:03.080 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:03.081+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:03.461 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:03.462+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:03.959 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:03.959+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:04.048 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:04.049+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:04.455 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:04.456+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:04.925 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:04.926+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:05.087 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:05.088+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:05.411 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:05.412+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:05.916 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:05.917+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:06.064 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:06.065+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:06.378 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:06.379+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:06.869 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:06.870+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:07.111 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:07.111+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:07.375 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:07.376+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:07.868 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:07.869+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:08.073 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:08.074+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:08.380 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:08.381+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:08.887 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:08.888+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:09.024 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:09.025+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:09.410 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:09.411+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:09.918 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:09.919+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:10.049 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:10.050+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:10.367 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:10.368+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:10.924 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:10.925+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:11.010 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:11.010+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:11.398 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:11.399+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:11.947 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:11.948+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:12.036 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:12.037+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:12.360 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:12.361+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:12.937 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:12.938+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:13.053 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:13.054+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:13.333 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:13.334+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:13.971 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:13.972+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:14.029 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:14.028+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:14.315 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:14.314+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:14.927 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:14.926+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:15.075 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:15.074+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:15.292 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:15.292+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:15.908 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:15.908+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:16.045 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:16.045+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:16.280 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:16.280+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:16.902 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:16.902+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:17.086 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:17.085+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:17.302 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:17.301+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:17.862 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:17.861+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:18.074 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:18.074+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:18.319 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:18.318+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:18.857 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:18.857+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:19.071 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:19.070+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:19.274 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:19.273+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:19.885 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:19.884+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:20.054 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:20.053+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:20.303 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:20.302+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:20.918 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:20.918+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:21.065 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:21.065+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:21.255 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:21.254+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:21.939 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:21.939+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:22.111 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:22.110+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:22.275 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:22.275+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:22.977 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:22.976+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:23.064 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:23.064+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:23.304 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:23.304+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:23.982 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:23.982+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:24.025 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:24.024+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:24.332 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:24.332+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:24.967 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:24.966+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:25.031 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:25.030+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:25.373 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:25.371+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:25.980 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:25.980+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:25.989 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:25.988+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:26.376 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:26.376+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:26.949 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:26.948+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:26.973 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:26.972+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:27.348 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:27.348+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:27.901 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:27.900+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:27.983 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:27.982+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:28.377 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:28.377+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:28.938 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:28.938+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:28.983 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:28.983+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:29.347 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:29.346+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:29.915 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:29.915+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:29.942 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:29.942+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:30.366 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:30.365+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:30.872 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:30.872+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:30.939 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:30.938+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:31.402 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:31.401+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:31.894 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:31.894+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:31.964 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:31.963+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:32.382 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:32.382+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:32.872 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:32.871+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:32.936 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:32.935+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:33.396 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:33.395+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:33.877 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:33.877+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:33.961 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:33.960+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:34.387 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:34.387+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:34.855 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:34.854+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:34.952 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:34.951+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:35.341 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:35.341+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:35.826 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:35.825+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:35.926 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:35.925+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:36.369 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:36.369+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:36.843 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:36.842+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:36.960 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:36.959+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:37.376 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:37.376+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:37.837 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:37.837+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:37.957 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:37.957+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:38.381 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:38.381+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:38.864 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:38.863+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:38.975 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:38.974+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:39.389 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:39.388+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:39.912 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:39.912+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:39.979 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:39.979+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:40.366 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:40.365+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:40.895 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:40.894+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:40.967 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:40.967+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:41.346 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:41.346+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:41.855 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:41.854+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:41.958 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:41.957+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:42.325 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:42.324+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:42.895 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:42.894+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:42.940 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:42.939+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:43.318 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:43.317+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:43.897 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:43.896+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:43.964 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:43.964+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:44.296 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:44.296+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:44.858 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:44.859+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:44.985 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:44.985+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:45.278 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:45.279+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:45.906 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:45.906+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:45.981 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:45.981+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:46.325 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:46.326+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:46.888 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:46.888+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:47.005 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:47.006+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:47.294 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:47.295+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:47.907 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:47.907+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:48.050 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:48.051+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:48.257 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:48.258+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:48.924 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:48.925+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:49.027 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:49.028+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:49.291 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:49.291+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:49.908 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:49.909+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:50.018 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:50.018+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:50.276 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:50.277+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:50.921 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:50.922+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:50.998 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:50.999+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:51.248 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:51.249+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:51.962 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:51.963+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:52.008 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:52.008+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:52.279 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:52.279+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:52.967 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:52.968+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:52.981 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:52.981+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:53.302 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:53.303+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:53.988 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:53.989+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:54.025 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:54.026+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:54.280 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:54.280+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:54.950 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:54.950+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:55.016 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:55.016+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:55.308 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:55.308+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:55.934 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:55.935+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:56.024 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:56.024+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:56.283 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:56.284+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:56.947 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:56.947+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:56.994 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:56.993+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:57.308 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:57.308+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:57.953 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:57.953+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:57.972 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:57.971+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:58.338 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:58.337+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:58.990 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:58.989+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:59.003 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:37:59.002+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:59.302 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:37:59.302+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:37:59.979 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:37:59.978+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:00.008 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:00.007+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:00.274 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:00.274+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:01.028 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:01.028+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:01.035 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:01.035+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:01.296 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:01.295+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:02.026 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:02.025+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:02.059 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:02.059+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:02.293 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:02.293+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:03.014 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:03.014+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:03.024 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:03.024+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:03.260 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:03.260+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:04.037 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:04.036+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:04.045 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:04.045+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:04.290 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:04.289+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:04.993 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:04.992+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:05.031 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:05.030+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:05.315 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:05.315+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:05.983 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:05.983+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:06.057 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:06.056+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:06.335 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:06.335+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:07.022 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:07.022+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:07.069 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:07.068+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:07.374 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:07.374+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:08.036 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:08.036+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:08.114 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:08.113+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:08.408 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:08.407+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:08.991 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:08.990+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:09.133 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:09.133+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:09.424 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:09.424+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:09.962 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:09.961+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:10.130 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:10.129+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:10.411 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:10.410+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:11.006 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:11.006+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:11.131 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:11.130+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:11.423 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:11.422+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:12.027 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:12.026+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:12.156 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:12.156+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:12.385 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:12.384+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:13.000 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:13.000+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:13.170 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:13.170+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:13.395 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:13.394+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:13.977 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:13.976+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:14.124 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:14.123+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:14.445 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:14.444+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:14.987 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:14.986+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:15.160 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:15.159+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:15.483 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:15.483+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:16.006 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:16.006+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:16.136 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:16.135+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:16.508 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:16.508+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:17.006 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:17.006+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:17.114 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:17.113+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:17.502 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:17.502+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:18.028 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:18.027+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:18.126 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:18.126+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:18.494 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:18.493+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:19.072 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:19.071+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:19.093 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:19.093+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:19.483 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:19.483+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:20.027 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:20.026+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:20.071 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:20.070+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:20.494 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:20.493+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:21.018 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:21.017+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:21.090 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:21.089+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:21.524 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:21.523+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:22.050 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:22.050+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:22.118 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:22.117+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:22.534 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:22.533+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:23.090 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:23.090+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:23.110 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:23.110+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:23.524 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:23.524+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:24.072 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:24.071+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:24.145 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:24.144+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:24.529 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:24.529+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:25.061 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:25.060+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:25.108 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:25.108+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:25.526 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:25.525+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:26.020 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:26.020+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:26.139 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:26.139+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:26.492 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:26.491+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:27.041 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:27.041+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:27.186 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:27.186+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:27.510 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:27.510+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:28.061 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:28.061+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:28.232 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:28.233+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:28.543 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:28.544+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:29.017 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:29.017+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:29.192 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:29.192+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:29.561 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:29.561+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:30.058 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:30.059+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:30.146 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:30.147+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:30.526 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:30.527+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:31.094 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:31.094+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:31.135 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:31.135+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:31.553 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:31.554+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:32.087 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:32.088+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:32.183 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:32.183+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:32.602 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:32.602+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:33.132 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:33.132+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:33.169 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:33.169+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:33.641 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:33.642+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:34.167 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:34.168+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:34.204 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:34.205+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:34.647 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:34.648+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:35.210 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:35.210+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:35.210 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:35.211+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:35.639 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:35.639+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:36.172 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:36.172+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:36.242 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:36.242+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:36.612 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:36.612+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:37.188 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:37.189+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:37.224 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:37.224+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:37.582 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:37.582+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:38.206 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:38.206+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:38.243 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:38.243+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:38.570 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:38.570+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:39.254 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:39.254+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:39.287 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:39.288+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:39.605 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:39.605+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:40.240 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:40.239+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:40.264 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:40.263+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:40.617 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:40.616+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:41.203 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:41.203+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:41.284 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:41.284+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:41.663 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:41.663+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:42.178 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:42.177+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:42.267 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:42.266+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:42.638 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:42.637+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:43.170 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:43.169+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:43.247 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:43.246+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:43.678 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:43.677+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:44.135 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:44.135+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:44.282 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:44.281+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:44.727 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:44.727+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:45.128 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:45.128+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:45.322 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:45.321+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:45.776 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:45.776+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:46.153 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:46.152+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:46.320 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:46.319+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:46.741 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:46.740+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:47.185 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:47.185+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:47.335 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:47.335+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:47.773 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:47.772+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:48.167 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:48.167+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:48.335 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:48.334+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:48.791 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:48.790+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:49.153 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:49.152+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:49.304 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:49.304+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:49.790 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:49.789+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:50.115 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:50.114+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:50.270 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:50.269+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:50.769 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:50.768+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:51.158 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:51.157+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:51.225 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:51.225+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:51.766 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:51.766+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:52.137 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:52.137+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:52.251 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:52.251+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:52.736 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:52.735+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:53.139 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:53.138+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:53.203 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:53.203+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:53.699 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:53.698+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:54.173 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:54.172+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:54.173 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:54.173+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:54.741 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:54.741+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:55.177 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:55.177+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:55.216 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:55.215+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:55.727 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:55.726+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:56.169 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:56.169+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:56.266 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:56.265+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:56.739 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:56.738+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:57.133 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:57.132+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:57.260 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:57.259+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:57.711 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:57.711+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:58.179 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:58.179+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:58.210 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:58.210+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:58.691 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:58.691+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:59.150 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:38:59.150+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:59.161 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:38:59.160+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:38:59.661 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:38:59.660+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:00.129 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:00.128+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:00.167 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:00.166+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:00.622 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:00.622+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:01.173 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:01.173+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:01.194 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:01.194+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:01.631 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:01.631+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:02.136 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:02.136+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:02.225 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:02.225+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:02.611 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:02.610+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:03.141 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:03.140+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:03.254 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:03.254+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:03.593 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:03.592+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:04.093 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:04.092+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:04.207 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:04.206+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:04.630 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:04.630+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:05.063 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:05.062+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:05.189 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:05.189+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:05.643 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:05.643+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:06.091 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:06.091+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:06.212 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:06.211+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:06.611 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:06.611+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:07.112 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:07.112+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:07.205 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:07.205+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:07.625 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:07.625+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:08.129 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:08.129+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:08.205 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:08.205+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:08.658 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:08.657+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:09.112 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:09.112+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:09.247 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:09.247+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:09.625 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:09.625+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:10.100 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:10.100+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:10.278 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:10.278+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:10.640 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:10.641+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:11.142 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:11.142+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:11.274 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:11.275+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:11.650 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:11.651+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:12.130 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:12.130+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:12.277 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:12.277+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:12.640 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:12.641+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:13.089 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:13.089+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:13.287 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:13.287+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:13.619 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:13.619+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:14.049 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:14.049+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:14.329 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:14.329+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:14.590 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:14.590+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:15.051 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:15.051+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:15.350 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:15.351+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:15.594 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:15.594+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:16.014 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:16.014+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:16.325 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:16.325+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:16.546 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:16.547+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:17.021 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:17.022+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:17.284 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:17.285+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:17.533 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:17.533+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:18.067 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:18.068+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:18.242 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:18.242+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:18.563 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:18.563+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:19.097 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:19.098+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:19.217 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:19.217+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:19.593 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:19.593+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:20.055 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:20.055+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:20.176 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:20.176+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:20.566 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:20.566+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:21.010 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:21.011+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:21.130 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:21.130+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:21.597 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:21.597+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:21.983 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:21.984+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:22.158 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:22.158+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:22.595 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:22.595+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:23.031 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:23.030+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:23.130 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:23.130+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:23.608 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:23.607+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:24.069 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:24.068+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:24.165 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:24.164+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:24.594 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:24.594+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:25.083 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:25.082+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:25.174 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:25.174+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:25.568 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:25.567+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:26.112 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:26.111+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:26.183 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:26.183+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:26.610 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:26.609+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:27.150 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:27.149+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:27.231 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:27.230+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:27.639 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:27.639+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:28.130 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:28.130+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:28.225 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:28.224+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:28.653 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:28.653+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:29.150 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:29.150+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:29.262 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:29.262+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:29.619 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:29.618+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:30.109 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:30.109+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:30.224 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:30.224+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:30.595 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:30.594+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:31.073 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:31.073+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:31.226 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:31.226+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:31.552 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:31.551+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:32.103 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:32.103+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:32.263 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:32.262+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:32.530 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:32.530+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:33.140 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:33.140+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:33.292 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:33.292+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:33.539 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:33.538+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:34.115 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:34.114+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:34.315 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:34.315+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:34.526 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:34.525+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:35.118 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:35.118+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:35.306 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:35.305+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:35.563 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:35.562+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:36.108 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:36.107+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:36.290 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:36.290+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:36.613 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:36.612+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:37.130 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:37.129+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:37.337 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:37.336+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:37.621 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:37.620+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:38.082 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:38.081+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:38.318 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:38.318+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:38.641 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:38.640+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:39.039 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:39.038+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:39.313 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:39.312+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:39.626 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:39.626+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:40.050 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:40.049+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:40.342 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:40.342+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:40.661 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:40.661+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:41.002 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:41.001+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:41.386 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:41.386+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:41.667 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:41.666+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:41.959 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:41.958+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:42.378 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:42.376+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:42.640 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:42.640+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:42.931 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:42.931+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:43.333 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:43.332+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:43.631 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:43.630+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:43.885 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:43.884+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:44.288 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:44.287+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:44.681 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:44.680+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:44.848 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:44.847+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:45.323 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:45.322+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:45.703 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:45.702+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:45.850 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:45.849+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:46.320 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:46.319+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:46.658 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:46.658+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:46.880 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:46.879+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:47.292 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:47.292+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:47.658 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:47.657+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:47.912 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:47.912+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:48.297 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:48.296+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:48.675 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:48.675+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:48.879 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:48.878+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:49.340 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:49.340+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:49.681 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:49.680+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:49.849 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:49.849+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:50.359 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:50.358+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:50.657 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:50.656+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:50.857 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:50.857+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:51.322 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:51.321+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:51.660 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:51.659+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:51.907 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:51.906+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:52.341 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:52.341+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:52.700 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:52.699+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:52.868 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:52.868+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:53.302 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:53.301+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:53.690 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:53.690+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:53.873 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:53.873+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:54.254 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:54.255+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:54.668 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:54.669+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:54.909 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:54.909+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:55.265 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:55.265+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:55.685 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:55.685+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:55.925 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:55.926+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:56.267 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:56.267+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:56.712 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:56.710+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:56.957 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:56.957+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:57.282 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:57.283+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:57.723 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:57.723+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:58.005 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:58.005+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:58.305 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:58.306+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:58.731 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:58.731+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:59.013 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:59.013+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:59.297 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:39:59.297+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:59.715 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:39:59.716+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:39:59.977 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:39:59.978+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:00.292 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:00.293+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:00.671 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:00.672+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:00.963 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:00.963+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:01.300 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:01.300+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:01.694 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:01.694+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:02.007 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:02.007+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:02.336 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:02.337+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:02.674 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:02.675+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:03.053 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:03.053+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:03.369 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:03.370+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:03.710 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:03.711+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:04.048 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:04.048+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:04.387 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:04.387+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:04.760 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:04.761+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:05.006 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:05.006+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:05.338 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:05.339+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:05.794 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:05.794+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:06.026 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:06.026+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:06.320 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:06.321+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:06.783 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:06.783+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:07.011 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:07.010+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:07.332 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:07.332+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:07.832 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:07.831+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:07.967 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:07.967+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:08.289 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:08.289+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:08.790 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:08.789+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:08.970 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:08.970+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:09.299 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:09.298+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:09.827 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:09.826+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:09.959 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:09.958+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:10.288 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:10.287+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:10.805 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:10.804+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:10.968 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:10.968+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:11.250 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:11.250+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:11.805 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:11.805+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:11.927 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:11.927+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:12.268 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:12.268+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:12.848 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:12.848+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:12.921 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:12.921+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:13.259 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:13.258+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:13.882 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:13.882+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:13.931 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:13.930+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:14.262 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:14.261+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:14.917 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:14.916+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:14.927 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:14.926+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:15.227 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:15.227+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:15.914 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:15.913+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:15.924 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:15.924+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:16.245 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:16.244+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:16.924 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:16.923+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:16.953 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:16.953+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:17.250 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:17.250+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:17.959 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:17.959+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:17.974 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:17.974+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:18.208 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:18.207+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:18.967 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:18.966+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:18.998 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:18.996+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:19.208 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:19.207+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:19.994 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:19.993+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:20.017 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:20.017+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:20.212 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:20.212+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:21.002 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:21.002+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:21.007 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:21.006+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:21.251 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:21.250+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:21.978 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:21.977+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:22.054 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:22.054+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:22.274 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:22.273+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:22.934 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:22.933+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:23.044 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:23.043+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:23.227 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:23.227+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:23.912 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:23.911+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:24.063 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:24.062+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:24.201 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:24.200+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:24.868 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:24.868+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:25.018 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:25.018+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:25.216 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:25.216+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:25.918 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:25.917+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:26.032 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:26.032+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:26.239 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:26.238+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:26.936 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:26.936+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:26.986 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:26.986+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:27.223 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:27.222+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:27.902 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:27.901+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:27.976 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:27.975+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:28.179 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:28.178+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:28.867 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:28.867+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:28.957 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:28.956+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:29.181 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:29.180+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:29.851 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:29.850+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:29.964 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:29.963+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:30.148 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:30.148+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:30.894 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:30.893+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:30.978 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:30.977+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:31.138 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:31.138+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:31.876 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:31.875+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:32.008 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:32.006+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:32.185 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:32.185+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:32.885 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:32.885+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:33.028 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:33.027+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:33.200 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:33.199+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:33.852 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:33.850+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:33.996 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:33.995+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:34.227 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:34.227+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:34.869 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:34.869+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:34.974 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:34.973+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:35.237 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:35.236+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:35.916 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:35.916+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:35.951 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:35.951+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:36.213 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:36.212+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:36.938 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:36.938+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:36.989 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:36.989+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:37.255 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:37.254+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:37.919 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:37.919+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:38.009 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:38.009+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:38.231 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:38.231+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:38.958 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:38.959+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:39.011 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:39.011+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:39.275 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:39.275+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:39.911 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:39.911+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:40.060 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:40.061+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:40.264 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:40.264+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:40.904 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:40.904+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:41.101 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:41.101+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:41.288 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:41.289+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:41.888 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:41.888+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:42.086 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:42.086+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:42.311 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:42.311+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:42.878 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:42.879+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:43.111 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:43.111+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:43.343 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:43.343+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:43.891 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:43.891+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:44.119 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:44.119+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:44.393 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:44.393+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:44.874 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:44.874+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:45.105 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:45.105+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:45.405 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:45.406+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:45.850 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:45.851+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:46.119 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:46.119+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:46.370 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:46.371+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:46.892 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:46.892+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:47.102 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:47.102+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:47.365 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:47.366+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:47.845 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:47.845+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:48.125 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:48.125+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:48.351 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:48.351+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:48.840 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:48.840+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:49.094 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:49.095+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:49.333 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:49.333+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:49.882 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:49.882+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:50.123 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:50.123+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:50.327 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:50.327+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:50.903 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:50.902+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:51.102 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:51.101+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:51.343 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:51.342+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:51.881 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:51.880+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:52.056 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:52.055+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:52.335 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:52.334+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:52.906 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:52.905+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:53.062 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:53.061+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:53.363 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:53.362+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:53.916 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:53.915+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:54.047 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:54.046+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:54.406 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:54.406+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:54.878 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:54.877+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:55.052 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:55.051+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:55.416 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:55.415+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:55.834 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:55.833+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:56.094 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:56.093+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:56.382 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:56.382+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:56.878 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:56.878+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:57.074 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:57.073+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:57.334 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:57.333+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:57.888 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:57.888+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:58.068 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:58.067+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:58.303 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:58.302+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:58.918 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:58.918+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:59.044 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:40:59.043+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:59.276 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:40:59.276+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:40:59.909 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:40:59.908+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:00.042 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:00.041+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:00.304 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:00.304+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:00.875 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:00.870+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:01.031 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:01.030+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:01.278 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:01.277+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:01.845 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:01.845+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:02.059 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:02.058+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:02.239 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:02.238+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:02.886 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:02.885+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:03.090 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:03.089+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:03.255 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:03.254+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:03.925 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:03.925+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:04.043 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:04.042+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:04.237 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:04.236+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:04.923 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:04.923+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:05.050 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:05.049+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:05.256 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:05.256+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:05.876 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:05.876+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:06.064 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:06.063+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:06.251 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:06.250+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:06.902 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:06.902+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:07.053 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:07.053+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:07.294 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:07.293+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:07.916 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:07.916+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:08.084 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:08.083+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:08.326 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:08.326+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:08.879 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:08.879+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:09.134 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:09.133+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:09.333 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:09.332+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:09.863 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:09.862+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:10.181 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:10.181+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:10.334 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:10.334+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:10.827 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:10.827+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:11.143 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:11.142+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:11.299 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:11.298+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:11.844 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:11.843+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:12.184 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:12.183+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:12.274 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:12.274+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:12.844 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:12.843+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:13.182 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:13.181+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:13.240 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:13.239+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:13.853 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:13.852+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:14.191 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:14.190+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:14.238 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:14.237+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:14.844 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:14.844+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:15.195 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:15.194+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:15.195 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:15.195+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:15.829 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:15.828+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:16.162 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:16.162+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:16.163 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:16.163+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:16.846 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:16.845+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:17.123 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:17.122+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:17.202 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:17.201+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:17.876 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:17.874+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:18.096 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:18.095+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:18.168 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:18.168+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:18.901 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:18.901+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:19.105 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:19.104+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:19.136 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:19.136+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:19.856 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:19.856+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:20.116 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:20.115+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:20.160 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:20.160+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:20.898 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:20.898+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:21.140 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:21.139+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:21.158 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:21.157+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:21.945 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:21.945+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:22.098 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:22.098+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:22.193 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:22.193+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:22.899 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:22.899+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:23.096 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:23.096+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:23.177 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:23.177+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:23.882 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:23.882+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:24.113 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:24.114+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:24.218 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:24.218+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:24.914 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:24.915+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:25.078 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:25.078+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:25.265 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:25.265+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:25.893 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:25.893+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:26.117 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:26.118+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:26.254 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:26.255+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:26.856 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:26.856+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:27.147 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:27.147+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:27.302 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:27.302+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:27.865 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:27.865+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:28.099 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:28.099+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:28.346 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:28.346+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:28.851 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:28.852+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:29.084 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:29.084+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:29.363 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:29.364+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:29.814 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:29.815+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:30.105 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:30.105+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:30.315 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:30.315+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:30.794 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:30.794+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:31.057 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:31.057+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:31.293 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:31.293+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:31.829 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:31.828+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:32.090 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:32.091+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:32.306 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:32.306+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:32.834 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:32.834+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:33.066 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:33.066+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:33.350 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:33.351+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:33.882 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:33.883+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:34.050 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:34.051+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:34.360 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:34.360+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:34.880 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:34.879+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:35.013 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:35.013+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:35.379 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:35.379+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:35.898 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:35.898+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:36.038 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:36.038+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:36.381 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:36.380+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:36.893 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:36.892+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:37.034 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:37.033+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:37.412 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:37.412+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:37.846 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:37.845+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:38.075 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:38.075+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:38.384 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:38.383+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:38.888 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:38.887+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:39.059 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:39.058+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:39.397 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:39.396+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:39.919 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:39.919+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:40.088 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:40.087+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:40.392 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:40.392+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:40.909 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:40.909+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:41.112 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:41.111+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:41.391 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:41.391+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:41.886 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:41.886+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:42.091 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:42.090+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:42.345 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:42.344+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:42.849 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:42.849+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:43.136 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:43.135+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:43.370 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:43.370+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:43.859 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:43.859+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:44.109 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:44.108+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:44.392 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:44.391+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:44.894 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:44.893+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:45.102 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:45.101+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:45.405 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:45.404+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:45.889 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:45.888+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:46.119 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:46.119+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:46.447 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:46.446+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:46.875 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:46.875+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:47.101 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:47.100+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:47.400 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:47.400+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:47.882 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:47.882+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:48.146 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:48.145+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:48.424 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:48.423+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:48.846 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:48.845+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:49.130 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:49.129+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:49.427 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:49.427+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:49.893 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:49.893+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:50.137 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:50.136+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:50.432 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:50.431+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:50.859 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:50.858+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:51.153 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:51.153+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:51.480 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:51.479+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:51.845 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:51.844+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:52.109 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:52.108+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:52.463 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:52.463+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:52.827 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:52.826+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:53.140 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:53.140+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:53.434 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:53.434+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:53.836 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:53.835+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:54.183 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:54.182+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:54.461 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:54.460+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:54.866 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:54.865+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:55.190 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:55.190+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:55.498 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:55.498+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:55.826 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:55.825+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:56.205 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:56.205+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:56.527 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:56.526+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:56.840 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:56.839+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:57.184 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:57.183+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:57.533 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:57.532+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:57.822 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:57.821+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:58.217 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:58.216+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:58.544 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:58.543+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:58.818 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:58.817+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:59.228 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:41:59.228+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:59.550 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:41:59.550+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:41:59.861 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:41:59.860+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:00.275 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:00.274+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:00.584 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:00.583+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:00.853 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:00.852+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:01.266 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:01.266+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:01.563 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:01.562+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:01.882 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:01.881+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:02.262 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:02.262+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:02.589 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:02.588+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:02.918 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:02.918+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:03.264 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:03.263+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:03.628 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:03.628+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:03.901 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:03.901+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:04.289 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:04.288+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:04.673 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:04.673+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:04.936 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:04.936+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:05.285 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:05.284+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:05.689 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:05.688+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:05.983 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:05.984+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:06.330 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:06.331+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:06.663 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:06.663+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:06.995 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:06.995+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:07.308 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:07.309+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:07.641 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:07.639+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:08.032 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:08.032+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:08.323 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:08.324+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:08.604 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:08.605+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:09.077 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:09.078+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:09.278 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:09.278+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:09.648 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:09.648+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:10.068 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:10.068+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:10.324 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:10.325+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:10.601 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:10.602+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:11.043 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:11.044+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:11.364 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:11.364+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:11.631 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:11.631+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:12.089 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:12.089+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:12.317 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:12.317+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:12.612 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:12.613+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:13.076 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:13.077+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:13.354 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:13.354+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:13.642 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:13.643+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:14.076 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:14.077+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:14.317 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:14.317+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:14.621 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:14.621+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:15.111 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:15.111+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:15.319 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:15.320+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:15.605 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:15.605+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:16.067 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:16.068+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:16.280 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:16.280+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:16.567 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:16.567+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:17.046 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:17.047+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:17.287 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:17.287+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:17.590 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:17.590+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:18.050 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:18.050+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:18.290 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:18.291+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:18.634 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:18.635+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:19.087 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:19.087+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:19.335 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:19.334+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:19.663 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:19.662+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:20.111 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:20.111+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:20.368 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:20.368+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:20.659 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:20.658+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:21.116 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:21.115+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:21.324 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:21.323+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:21.629 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:21.628+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:22.081 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:22.079+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:22.299 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:22.298+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:22.613 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:22.613+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:23.082 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:23.081+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:23.329 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:23.329+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:23.575 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:23.575+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:24.085 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:24.084+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:24.354 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:24.353+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:24.542 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:24.541+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:25.079 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:25.078+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:25.379 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:25.378+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:25.546 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:25.546+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:26.127 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:26.126+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:26.334 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:26.333+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:26.560 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:26.559+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:27.094 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:27.094+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:27.318 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:27.318+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:27.566 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:27.566+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:28.055 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:28.055+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:28.329 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:28.328+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:28.600 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:28.599+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:29.023 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:29.023+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:29.365 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:29.364+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:29.630 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:29.629+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:30.060 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:30.059+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:30.376 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:30.376+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:30.654 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:30.653+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:31.046 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:31.045+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:31.358 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:31.357+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:31.635 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:31.634+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:32.002 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:32.002+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:32.386 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:32.385+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:32.622 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:32.622+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:32.996 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:32.996+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:33.420 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:33.420+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:33.656 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:33.655+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:33.977 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:33.977+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:34.410 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:34.409+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:34.690 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:34.690+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:34.976 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:34.975+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:35.391 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:35.390+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:35.698 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:35.697+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:35.998 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:35.997+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:36.388 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:36.388+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:36.700 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:36.699+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:36.958 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:36.958+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:37.424 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:37.423+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:37.730 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:37.730+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:37.950 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:37.950+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:38.402 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:38.402+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:38.765 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:38.765+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:38.934 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:38.933+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:39.380 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:39.380+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:39.718 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:39.717+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:39.979 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:39.979+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:40.401 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:40.400+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:40.672 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:40.671+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:40.969 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:40.968+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:41.419 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:41.418+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:41.638 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:41.638+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:41.920 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:41.919+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:42.403 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:42.402+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:42.680 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:42.679+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:42.881 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:42.881+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:43.395 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:43.394+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:43.661 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:43.660+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:43.859 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:43.858+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:44.380 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:44.379+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:44.630 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:44.629+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:44.825 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:44.824+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:45.407 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:45.406+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:45.610 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:45.610+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:45.833 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:45.832+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:46.387 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:46.386+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:46.620 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:46.620+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:46.854 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:46.853+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:47.370 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:47.369+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:47.586 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:47.586+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:47.872 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:47.872+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:48.388 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:48.388+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:48.598 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:48.598+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:48.890 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:48.890+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:49.366 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:49.366+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:49.594 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:49.593+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:49.847 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:49.847+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:50.370 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:50.370+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:50.587 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:50.588+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:50.884 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:50.885+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:51.352 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:51.353+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:51.582 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:51.583+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:51.849 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:51.849+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:52.359 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:52.360+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:52.580 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:52.580+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:52.852 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:52.852+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:53.357 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:53.358+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:53.544 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:53.544+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:53.884 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:53.885+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:54.375 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:54.374+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:54.507 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:54.508+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:54.909 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:54.910+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:55.375 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:55.375+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:55.525 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:55.526+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:55.910 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:55.911+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:56.413 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:56.414+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:56.530 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:56.530+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:56.951 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:56.951+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:57.372 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:57.373+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:57.546 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:57.546+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:57.960 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:57.960+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:58.406 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:58.406+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:58.527 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:58.527+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:58.933 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:58.933+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:59.428 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:42:59.429+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:59.563 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:42:59.563+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:42:59.952 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:42:59.953+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:00.454 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:00.454+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:00.542 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:00.543+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:00.994 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:00.994+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:01.431 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:01.431+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:01.506 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:01.506+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:01.982 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:01.983+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:02.436 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:02.436+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:02.464 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:02.465+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:03.003 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:03.002+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:03.395 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:03.394+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:03.425 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:03.425+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:03.977 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:03.976+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:04.392 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:04.391+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:04.455 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:04.454+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:05.024 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:05.024+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:05.429 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:05.429+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:05.437 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:05.435+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:06.027 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:06.026+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:06.390 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:06.390+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:06.393 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:06.393+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:07.069 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:07.067+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:07.343 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:07.342+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:07.405 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:07.404+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:08.044 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:08.043+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:08.340 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:08.339+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:08.408 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:08.407+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:09.011 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:09.011+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:09.362 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:09.362+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:09.401 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:09.401+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:10.036 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:10.036+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:10.376 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:10.376+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:10.403 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:10.403+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:11.077 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:11.076+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:11.373 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:11.372+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:11.404 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:11.403+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:12.123 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:12.122+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:12.385 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:12.384+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:12.401 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:12.401+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:13.152 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:13.152+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:13.344 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:13.343+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:13.377 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:13.376+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:14.169 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:14.168+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:14.356 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:14.355+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:14.356 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:14.355+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:15.210 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:15.210+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:15.323 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:15.322+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:15.340 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:15.339+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:16.180 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:16.177+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:16.281 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:16.280+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:16.313 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:16.313+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:17.133 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:17.132+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:17.282 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:17.281+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:17.297 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:17.296+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:18.166 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:18.162+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:18.260 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:18.258+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:18.262 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:18.261+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:19.157 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:19.157+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:19.251 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:19.250+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:19.258 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:19.257+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:20.136 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:20.135+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:20.225 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:20.224+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:20.270 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:20.269+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:21.097 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:21.096+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:21.202 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:21.201+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:21.264 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:21.263+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:22.101 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:22.100+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:22.241 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:22.240+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:22.310 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:22.310+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:23.066 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:23.066+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:23.273 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:23.273+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:23.344 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:23.343+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:24.034 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:24.034+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:24.251 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:24.250+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:24.329 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:24.328+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:25.035 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:25.034+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:25.287 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:25.286+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:25.343 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:25.342+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:25.986 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:25.985+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:26.298 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:26.297+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:26.303 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:26.302+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:26.997 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:26.996+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:27.291 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:27.290+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:27.302 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:27.302+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:27.977 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:27.977+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:28.247 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:28.246+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:28.264 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:28.263+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:28.953 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:28.953+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:29.204 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:29.204+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:29.241 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:29.241+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:29.923 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:29.922+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:30.166 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:30.165+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:30.239 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:30.238+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:30.878 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:30.877+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:31.141 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:31.140+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:31.255 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:31.254+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:31.924 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:31.923+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:32.161 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:32.160+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:32.246 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:32.246+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:32.966 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:32.966+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:33.170 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:33.170+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:33.233 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:33.232+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:34.004 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:34.004+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:34.217 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:34.217+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:34.257 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:34.258+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:35.023 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:35.023+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:35.249 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:35.246+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:35.264 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:35.263+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:35.976 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:35.977+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:36.290 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:36.290+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:36.295 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:36.296+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:36.937 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:36.938+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:37.321 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:37.321+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:37.329 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:37.330+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:37.954 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:37.954+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:38.287 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:38.287+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:38.321 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:38.321+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:38.973 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:38.973+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:39.292 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:39.292+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:39.369 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:39.369+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:39.983 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:39.984+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:40.283 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:40.283+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:40.399 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:40.400+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:41.015 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:41.014+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:41.291 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:41.291+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:41.372 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:41.372+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:42.023 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:42.023+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:42.244 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:42.245+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:42.363 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:42.363+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:42.975 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:42.975+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:43.203 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:43.203+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:43.347 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:43.348+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:43.930 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:43.931+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:44.205 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:44.206+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:44.320 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:44.320+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:44.905 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:44.905+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:45.205 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:45.205+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:45.340 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:45.340+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:45.954 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:45.954+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:46.199 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:46.200+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:46.376 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:46.377+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:46.983 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:46.982+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:47.210 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:47.210+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:47.415 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:47.415+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:47.940 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:47.939+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:48.193 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:48.193+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:48.442 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:48.441+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:48.946 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:48.934+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:49.203 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:49.202+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:49.490 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:49.489+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:49.981 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:49.980+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:50.186 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:50.185+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:50.448 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:50.447+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:50.994 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:50.994+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:51.228 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:51.227+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:51.489 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:51.489+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:52.011 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:52.010+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:52.273 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:52.273+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:52.495 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:52.495+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:52.974 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:52.973+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:53.241 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:53.240+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:53.491 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:53.491+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:53.955 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:53.955+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:54.198 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:54.197+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:54.466 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:54.466+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:55.004 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:55.003+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:55.179 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:55.179+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:55.481 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:55.480+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:56.004 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:56.004+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:56.160 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:56.159+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:56.512 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:56.512+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:57.032 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:57.031+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:57.179 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:57.179+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:57.516 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:57.514+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:58.041 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:58.040+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:58.228 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:58.228+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:58.512 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:58.512+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:59.020 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:43:59.020+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:59.207 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:43:59.207+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:43:59.497 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:43:59.496+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:00.006 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:00.006+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:00.232 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:00.231+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:00.468 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:00.468+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:00.989 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:00.988+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:01.229 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:01.228+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:01.501 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:01.500+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:01.950 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:01.950+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:02.183 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:02.182+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:02.494 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:02.493+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:02.917 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:02.917+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:03.171 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:03.171+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:03.544 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:03.543+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:03.872 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:03.872+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:04.156 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:04.155+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:04.505 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:04.505+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:04.894 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:04.893+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:05.175 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:05.175+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:05.503 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:05.502+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:05.885 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:05.884+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:06.133 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:06.133+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:06.461 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:06.461+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:06.914 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:06.913+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:07.104 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:07.103+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:07.451 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:07.450+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:07.911 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:07.910+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:08.068 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:08.067+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:08.483 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:08.482+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:08.941 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:08.940+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:09.037 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:09.036+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:09.444 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:09.443+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:09.981 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:09.981+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:10.001 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:10.000+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:10.447 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:10.446+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:10.982 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:10.981+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:10.990 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:10.990+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:11.492 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:11.492+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:11.960 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:11.959+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:11.966 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:11.965+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:12.445 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:12.444+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:12.959 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:12.958+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:12.995 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:12.995+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:13.396 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:13.396+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:13.982 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:13.982+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:14.039 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:14.038+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:14.426 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:14.425+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:15.030 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:15.029+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:15.038 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:15.037+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:15.383 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:15.382+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:16.034 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:16.033+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:16.059 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:16.059+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:16.340 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:16.340+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:17.021 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:17.021+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:17.108 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:17.108+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:17.346 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:17.345+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:18.010 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:18.009+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:18.156 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:18.157+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:18.369 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:18.370+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:18.967 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:18.967+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:19.196 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:19.197+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:19.364 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:19.364+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:19.989 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:19.989+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:20.204 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:20.204+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:20.401 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:20.400+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:21.039 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:21.039+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:21.244 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:21.245+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:21.357 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:21.358+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:22.061 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:22.061+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:22.250 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:22.250+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:22.396 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:22.396+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:23.063 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:23.064+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:23.290 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:23.290+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:23.374 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:23.375+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:24.032 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:24.032+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:24.244 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:24.245+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:24.374 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:24.374+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:25.045 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:25.045+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:25.248 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:25.249+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:25.334 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:25.335+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:26.061 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:26.061+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:26.211 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:26.211+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:26.322 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:26.323+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:27.096 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:27.096+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:27.231 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:27.232+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:27.367 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:27.367+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:28.096 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:28.096+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:28.201 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:28.201+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:28.394 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:28.395+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:29.067 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:29.067+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:29.192 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:29.192+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:29.434 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:29.435+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:30.047 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:30.047+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:30.223 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:30.224+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:30.427 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:30.427+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:31.002 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:31.002+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:31.273 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:31.272+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:31.478 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:31.477+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:32.035 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:32.034+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:32.310 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:32.309+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:32.521 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:32.520+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:33.033 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:33.032+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:33.359 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:33.359+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:33.480 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:33.479+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:34.055 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:34.055+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:34.369 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:34.369+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:34.447 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:34.446+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:35.080 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:35.079+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:35.348 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:35.347+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:35.403 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:35.402+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:36.033 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:36.032+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:36.332 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:36.332+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:36.443 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:36.442+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:37.067 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:37.066+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:37.287 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:37.287+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:37.432 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:37.432+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:38.030 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:38.030+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:38.318 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:38.317+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:38.407 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:38.406+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:39.062 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:39.062+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:39.300 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:39.299+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:39.386 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:39.385+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:40.014 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:40.013+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:40.349 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:40.349+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:40.398 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:40.397+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:41.032 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:41.031+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:41.353 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:41.352+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:41.436 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:41.435+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:42.044 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:42.043+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:42.378 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:42.378+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:42.424 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:42.423+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:42.998 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:42.998+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:43.406 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:43.406+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:43.443 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:43.443+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:44.000 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:44.000+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:44.369 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:44.369+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:44.402 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:44.402+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:45.016 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:45.016+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:45.376 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:45.376+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:45.377 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:45.377+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:45.973 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:45.972+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:46.379 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:46.377+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:46.379 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:46.378+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:46.933 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:46.932+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:47.369 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:47.368+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:47.420 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:47.419+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:47.914 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:47.913+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:48.386 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:48.386+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:48.416 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:48.415+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:48.881 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:48.880+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:49.393 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:49.392+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:49.436 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:49.436+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:49.894 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:49.894+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:50.364 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:50.364+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:50.398 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:50.397+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:50.929 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:50.928+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:51.378 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:51.377+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:51.447 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:51.446+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:51.965 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:51.964+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:52.350 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:52.349+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:52.479 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:52.478+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:52.969 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:52.969+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:53.353 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:53.353+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:53.494 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:53.494+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:53.960 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:53.959+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:54.342 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:54.341+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:54.542 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:54.541+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:54.994 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:54.993+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:55.363 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:55.362+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:55.556 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:55.553+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:55.949 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:55.948+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:56.363 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:56.362+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:56.584 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:56.583+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:56.976 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:56.976+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:57.368 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:57.368+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:57.562 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:57.561+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:58.023 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:58.022+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:58.397 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:58.396+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:58.513 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:58.512+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:59.063 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:44:59.062+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:59.445 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:44:59.444+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:44:59.511 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:44:59.511+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:00.090 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:00.089+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:00.425 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:00.425+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:00.557 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:00.556+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:01.128 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:01.127+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:01.410 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:01.409+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:01.510 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:01.509+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:02.085 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:02.084+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:02.450 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:02.449+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:02.550 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:02.550+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:03.090 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:03.090+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:03.456 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:03.456+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:03.568 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:03.569+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:04.114 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:04.114+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:04.412 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:04.412+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:04.599 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:04.599+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:05.090 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:05.090+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:05.396 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:05.397+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:05.582 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:05.583+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:06.108 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:06.108+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:06.357 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:06.358+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:06.582 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:06.582+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:07.104 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:07.104+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:07.364 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:07.364+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:07.543 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:07.544+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:08.148 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:08.149+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:08.379 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:08.380+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:08.572 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:08.572+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:09.182 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:09.183+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:09.393 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:09.393+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:09.570 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:09.570+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:10.164 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:10.165+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:10.348 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:10.348+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:10.581 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:10.581+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:11.162 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:11.162+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:11.389 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:11.389+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:11.595 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:11.595+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:12.157 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:12.157+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:12.350 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:12.350+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:12.608 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:12.608+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:13.119 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:13.120+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:13.364 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:13.364+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:13.601 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:13.601+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:14.172 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:14.170+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:14.352 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:14.352+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:14.640 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:14.641+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:15.210 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:15.210+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:15.382 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:15.383+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:15.642 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:15.642+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:16.163 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:16.162+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:16.411 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:16.410+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:16.640 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:16.639+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:17.163 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:17.162+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:17.378 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:17.377+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:17.607 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:17.607+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:18.207 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:18.207+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:18.337 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:18.336+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:18.579 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:18.578+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:19.183 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:19.183+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:19.350 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:19.349+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:19.544 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:19.544+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:20.189 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:20.188+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:20.338 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:20.337+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:20.551 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:20.551+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:21.207 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:21.206+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:21.384 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:21.384+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:21.520 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:21.519+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:22.239 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:22.238+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:22.389 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:22.388+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:22.569 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:22.568+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:23.211 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:23.210+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:23.367 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:23.366+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:23.526 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:23.525+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:24.173 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:24.172+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:24.379 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:24.378+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:24.515 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:24.514+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:25.169 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:25.168+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:25.417 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:25.416+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:25.563 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:25.563+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:26.211 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:26.210+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:26.461 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:26.460+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:26.593 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:26.593+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:27.170 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:27.169+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:27.498 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:27.497+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:27.558 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:27.557+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:28.203 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:28.202+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:28.450 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:28.449+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:28.547 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:28.546+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:29.195 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:29.195+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:29.497 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:29.496+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:29.531 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:29.531+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:30.168 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:30.167+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:30.495 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:30.494+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:30.535 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:30.535+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:31.172 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:31.171+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:31.532 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:31.532+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:31.552 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:31.547+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:32.189 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:32.188+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:32.512 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:32.512+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:32.526 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:32.526+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:33.162 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:33.161+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:33.481 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:33.481+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:33.545 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:33.545+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:34.182 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:34.181+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:34.436 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:34.436+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:34.509 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:34.508+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:35.204 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:35.203+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:35.464 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:35.463+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:35.474 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:35.474+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:36.155 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:36.155+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:36.496 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:36.495+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:36.521 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:36.521+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:37.149 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:37.148+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:37.510 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:37.510+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:37.512 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:37.512+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:38.136 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:38.136+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:38.467 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:38.466+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:38.561 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:38.560+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:39.092 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:39.091+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:39.466 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:39.465+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:39.604 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:39.603+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:40.123 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:40.122+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:40.442 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:40.441+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:40.579 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:40.578+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:41.153 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:41.153+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:41.446 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:41.445+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:41.566 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:41.565+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:42.107 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:42.106+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:42.427 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:42.426+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:42.605 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:42.604+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:43.092 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:43.091+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:43.387 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:43.386+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:43.646 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:43.645+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:44.127 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:44.126+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:44.425 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:44.424+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:44.657 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:44.656+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:45.133 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:45.132+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:45.405 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:45.405+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:45.678 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:45.678+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:46.116 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:46.115+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:46.449 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:46.449+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:46.648 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:46.648+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:47.066 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:47.066+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:47.479 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:47.479+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:47.691 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:47.691+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:48.031 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:48.031+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:48.471 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:48.471+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:48.675 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:48.676+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:49.030 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:49.030+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:49.442 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:49.443+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:49.687 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:49.687+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:50.027 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:50.027+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:50.486 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:50.486+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:50.702 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:50.702+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:51.049 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:51.048+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:51.478 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:51.478+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:51.659 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:51.659+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:52.015 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:52.016+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:52.444 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:52.444+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:52.630 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:52.630+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:53.033 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:53.033+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:53.435 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:53.435+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:53.670 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:53.670+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:53.991 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:53.992+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:54.422 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:54.422+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:54.705 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:54.705+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:54.978 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:54.979+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:55.414 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:55.415+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:55.701 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:55.701+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:55.936 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:55.937+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:56.423 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:56.423+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:56.685 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:56.686+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:56.971 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:56.971+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:57.430 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:57.431+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:57.699 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:57.700+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:57.943 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:57.943+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:58.452 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:58.452+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:58.682 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:58.682+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:58.899 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:58.899+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:59.452 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:45:59.453+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:59.683 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:45:59.683+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:45:59.926 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:45:59.926+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:00.496 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:00.495+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:00.665 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:00.664+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:00.920 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:00.919+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:01.509 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:01.508+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:01.618 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:01.617+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:01.882 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:01.881+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:02.504 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:02.504+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:02.652 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:02.651+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:02.851 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:02.851+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:03.532 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:03.531+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:03.698 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:03.698+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:03.859 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:03.858+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:04.541 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:04.540+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:04.693 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:04.692+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:04.827 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:04.826+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:05.529 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:05.526+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:05.697 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:05.696+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:05.791 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:05.791+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:06.523 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:06.522+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:06.649 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:06.648+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:06.794 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:06.794+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:07.547 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:07.546+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:07.662 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:07.661+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:07.789 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:07.788+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:08.532 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:08.531+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:08.666 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:08.666+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:08.773 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:08.772+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:09.561 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:09.561+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:09.644 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:09.644+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:09.821 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:09.821+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:10.606 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:10.605+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:10.674 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:10.674+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:10.802 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:10.801+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:11.628 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:11.627+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:11.653 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:11.653+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:11.804 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:11.803+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:12.601 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:12.600+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:12.683 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:12.683+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:12.763 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:12.762+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:13.601 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:13.600+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:13.671 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:13.671+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:13.743 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:13.742+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:14.609 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:14.608+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:14.638 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:14.638+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:14.703 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:14.703+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:15.596 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:15.595+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:15.623 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:15.622+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:15.660 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:15.659+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:16.556 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:16.555+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:16.657 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:16.656+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:16.697 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:16.697+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:17.582 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:17.581+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:17.634 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:17.633+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:17.730 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:17.729+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:18.612 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:18.612+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:18.626 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:18.626+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:18.751 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:18.751+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:19.574 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:19.573+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:19.601 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:19.601+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:19.755 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:19.754+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:20.555 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:20.555+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:20.585 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:20.584+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:20.729 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:20.728+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:21.515 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:21.515+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:21.610 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:21.609+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:21.681 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:21.680+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:22.565 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:22.554+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:22.660 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:22.659+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:22.661 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:22.661+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:23.526 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:23.526+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:23.703 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:23.702+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:23.708 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:23.708+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:24.563 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:24.562+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:24.706 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:24.705+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:24.731 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:24.730+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:25.560 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:25.559+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:25.711 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:25.711+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:25.712 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:25.711+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:26.519 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:26.519+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:26.683 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:26.682+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:26.723 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:26.723+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:27.482 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:27.481+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:27.686 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:27.685+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:27.689 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:27.688+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:28.504 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:28.503+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:28.683 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:28.683+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:28.728 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:28.727+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:29.539 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:29.539+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:29.663 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:29.663+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:29.737 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:29.737+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:30.492 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:30.492+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:30.656 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:30.655+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:30.737 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:30.736+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:31.465 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:31.465+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:31.631 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:31.631+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:31.702 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:31.702+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:32.429 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:32.429+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:32.613 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:32.613+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:32.752 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:32.752+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:33.393 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:33.393+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:33.592 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:33.593+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:33.710 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:33.710+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:34.383 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:34.384+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:34.612 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:34.612+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:34.741 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:34.742+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:35.410 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:35.410+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:35.650 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:35.650+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:35.789 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:35.790+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:36.457 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:36.457+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:36.656 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:36.656+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:36.768 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:36.768+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:37.426 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:37.426+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:37.654 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:37.654+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:37.787 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:37.787+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:38.377 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:38.377+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:38.630 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:38.630+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:38.759 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:38.759+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:39.339 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:39.339+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:39.622 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:39.623+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:39.768 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:39.768+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:40.380 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:40.380+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:40.617 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:40.617+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:40.777 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:40.777+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:41.407 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:41.407+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:41.570 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:41.571+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:41.738 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:41.738+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:42.396 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:42.397+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:42.605 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:42.606+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:42.705 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:42.705+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:43.442 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:43.443+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:43.637 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:43.637+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:43.722 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:43.723+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:44.447 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:44.447+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:44.666 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:44.665+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:44.702 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:44.701+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:45.490 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:45.489+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:45.692 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:45.691+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:45.701 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:45.701+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:46.452 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:46.452+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:46.679 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:46.679+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:46.743 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:46.743+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:47.419 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:47.418+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:47.685 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:47.684+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:47.722 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:47.721+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:48.415 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:48.415+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:48.733 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:48.732+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:48.744 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:48.744+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:49.442 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:49.441+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:49.727 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:49.726+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:49.744 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:49.744+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:50.475 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:50.475+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:50.680 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:50.680+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:50.711 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:50.711+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:51.490 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:51.490+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:51.712 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:51.711+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:51.755 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:51.755+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:52.539 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:52.538+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:52.732 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:52.732+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:52.802 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:52.801+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:53.584 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:53.584+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:53.754 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:53.753+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:53.783 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:53.782+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:54.601 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:54.601+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:54.737 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:54.736+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:54.763 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:54.762+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:55.559 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:55.558+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:55.734 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:55.733+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:55.753 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:55.753+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:56.535 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:56.534+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:56.743 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:56.742+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:56.764 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:56.764+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:57.566 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:57.565+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:57.761 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:57.760+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:57.791 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:57.790+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:58.594 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:58.590+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:58.795 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:58.792+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:58.800 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:58.799+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:59.599 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:46:59.598+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:59.757 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:46:59.756+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:46:59.758 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:46:59.758+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:00.564 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:00.563+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:00.788 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:00.787+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:00.803 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:00.802+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:01.569 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:01.568+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:01.756 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:01.755+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:01.799 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:01.798+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:02.522 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:02.521+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:02.734 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:02.733+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:02.812 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:02.811+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:03.501 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:03.501+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:03.733 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:03.732+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:03.769 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:03.768+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:04.516 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:04.516+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:04.763 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:04.762+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:04.764 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:04.764+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:05.476 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:05.475+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:05.754 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:05.753+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:05.780 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:05.779+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:06.498 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:06.498+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:06.799 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:06.798+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:06.803 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:06.802+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:07.499 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:07.498+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:07.760 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:07.760+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:07.785 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:07.784+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:08.490 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:08.489+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:08.745 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:08.744+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:08.761 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:08.760+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:09.456 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:09.455+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:09.768 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:09.768+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:09.794 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:09.793+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:10.419 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:10.419+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:10.743 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:10.742+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:10.794 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:10.793+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:11.456 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:11.455+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:11.717 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:11.716+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:11.760 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:11.759+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:12.417 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:12.416+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:12.676 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:12.676+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:12.763 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:12.762+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:13.418 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:13.417+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:13.654 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:13.653+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:13.807 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:13.806+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:14.405 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:14.404+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:14.678 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:14.677+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:14.819 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:14.819+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:15.437 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:15.437+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:15.705 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:15.704+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:15.772 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:15.772+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:16.478 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:16.479+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:16.679 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:16.679+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:16.737 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:16.737+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:17.455 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:17.455+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:17.700 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:17.700+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:17.720 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:17.720+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:18.469 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:18.469+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:18.659 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:18.660+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:18.751 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:18.751+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:19.446 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:19.447+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:19.624 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:19.624+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:19.799 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:19.799+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:20.492 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:20.492+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:20.663 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:20.664+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:20.763 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:20.763+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:21.491 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:21.491+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:21.671 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:21.671+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:21.731 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:21.732+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:22.488 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:22.488+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:22.655 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:22.656+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:22.711 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:22.711+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:23.518 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:23.518+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:23.690 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:23.690+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:23.758 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:23.759+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:24.526 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:24.527+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:24.702 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:24.702+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:24.807 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:24.808+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:25.517 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:25.517+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:25.675 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:25.676+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:25.824 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:25.824+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:26.553 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:26.553+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:26.690 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:26.690+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:26.787 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:26.787+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:27.537 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:27.537+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:27.687 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:27.687+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:27.803 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:27.803+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:28.541 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:28.541+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:28.644 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:28.644+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:28.846 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:28.847+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:29.592 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:29.590+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:29.684 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:29.683+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:29.846 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:29.846+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:30.609 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:30.608+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:30.676 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:30.676+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:30.852 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:30.851+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:31.646 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:31.645+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:31.701 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:31.700+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:31.844 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:31.844+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:32.636 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:32.636+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:32.750 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:32.749+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:32.889 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:32.888+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:33.591 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:33.590+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:33.795 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:33.794+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:33.860 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:33.859+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:34.599 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:34.599+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:34.757 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:34.756+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:34.827 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:34.826+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:35.637 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:35.636+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:35.715 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:35.714+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:35.825 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:35.824+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:36.589 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:36.588+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:36.716 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:36.715+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:36.872 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:36.871+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:37.626 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:37.626+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:37.713 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:37.712+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:37.853 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:37.853+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:38.600 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:38.599+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:38.701 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:38.700+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:38.819 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:38.819+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:39.614 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:39.614+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:39.685 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:39.685+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:39.833 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:39.830+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:40.647 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:40.646+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:40.677 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:40.676+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:40.820 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:40.819+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:41.696 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:41.695+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:41.696 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:41.695+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:41.789 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:41.788+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:42.660 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:42.659+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:42.676 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:42.675+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:42.740 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:42.739+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:43.645 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:43.645+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:43.649 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:43.649+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:43.791 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:43.790+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:44.612 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:44.611+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:44.689 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:44.689+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:44.794 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:44.793+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:45.609 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:45.609+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:45.691 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:45.690+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:45.841 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:45.840+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:46.650 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:46.649+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:46.686 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:46.686+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:46.844 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:46.843+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:47.617 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:47.613+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:47.680 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:47.679+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:47.840 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:47.839+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:48.590 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:48.589+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:48.656 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:48.655+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:48.825 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:48.824+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:49.620 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:49.620+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:49.667 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:49.666+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:49.870 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:49.869+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:50.654 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:50.653+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:50.702 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:50.701+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:50.849 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:50.848+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:51.626 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:51.625+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:51.731 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:51.731+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:51.814 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:51.813+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:52.624 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:52.624+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:52.759 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:52.758+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:52.850 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:52.849+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:53.659 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:53.659+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:53.761 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:53.761+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:53.817 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:53.816+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:54.698 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:54.698+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:54.795 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:54.794+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:54.863 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:54.862+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:55.680 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:55.680+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:55.823 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:55.822+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:55.899 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:55.899+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:56.670 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:56.669+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:56.794 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:56.793+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:56.864 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:56.863+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:57.682 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:57.681+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:57.800 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:57.799+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:57.832 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:57.831+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:58.652 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:58.652+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:58.760 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:58.760+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:58.865 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:58.862+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:59.626 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:47:59.625+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:59.806 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:47:59.806+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:47:59.877 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:47:59.876+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:00.612 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:00.612+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:00.774 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:00.774+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:00.906 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:00.906+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:01.630 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:01.630+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:01.767 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:01.768+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:01.939 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:01.939+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:02.664 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:02.665+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:02.813 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:02.814+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:02.990 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:02.989+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:03.707 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:03.707+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:03.829 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:03.829+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:03.972 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:03.972+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:04.666 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:04.666+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:04.858 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:04.858+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:05.013 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:05.014+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:05.656 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:05.656+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:05.825 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:05.825+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:06.032 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:06.032+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:06.659 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:06.660+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:06.812 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:06.812+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:07.019 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:07.020+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:07.656 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:07.655+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:07.786 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:07.787+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:07.989 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:07.989+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:08.692 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:08.692+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:08.790 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:08.790+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:09.011 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:09.011+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:09.661 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:09.662+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:09.750 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:09.750+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:10.043 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:10.043+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:10.631 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:10.632+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:10.744 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:10.743+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:11.040 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:11.040+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:11.665 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:11.666+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:11.694 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:11.694+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:12.045 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:12.045+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:12.621 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:12.622+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:12.657 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:12.658+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:13.061 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:13.062+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:13.583 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:13.583+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:13.695 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:13.694+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:14.023 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:14.022+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:14.625 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:14.625+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:14.696 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:14.695+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:14.989 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:14.988+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:15.633 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:15.633+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:15.671 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:15.671+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:15.949 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:15.948+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:16.595 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:16.594+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:16.721 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:16.721+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:16.952 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:16.951+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:17.603 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:17.602+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:17.746 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:17.746+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:17.985 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:17.984+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:18.572 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:18.571+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:18.712 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:18.711+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:18.993 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:18.992+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:19.584 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:19.583+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:19.678 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:19.678+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:19.964 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:19.963+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:20.543 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:20.543+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:20.646 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:20.645+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:20.932 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:20.931+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:21.547 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:21.546+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:21.602 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:21.602+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:21.895 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:21.894+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:22.503 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:22.502+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:22.576 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:22.575+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:22.856 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:22.855+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:23.455 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:23.454+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:23.549 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:23.549+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:23.887 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:23.886+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:24.411 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:24.411+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:24.530 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:24.530+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:24.913 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:24.912+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:25.411 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:25.411+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:25.491 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:25.490+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:25.889 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:25.888+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:26.458 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:26.457+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:26.481 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:26.480+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:26.848 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:26.847+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:27.485 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:27.484+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:27.526 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:27.525+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:27.819 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:27.818+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:28.508 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:28.507+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:28.562 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:28.562+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:28.815 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:28.815+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:29.504 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:29.503+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:29.556 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:29.555+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:29.804 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:29.804+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:30.459 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:30.458+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:30.537 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:30.536+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:30.845 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:30.845+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:31.481 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:31.480+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:31.508 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:31.507+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:31.800 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:31.800+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:32.477 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:32.477+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:32.480 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:32.479+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:32.776 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:32.774+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:33.442 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:33.441+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:33.467 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:33.466+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:33.739 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:33.738+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:34.420 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:34.419+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:34.471 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:34.470+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:34.713 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:34.711+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:35.432 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:35.431+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:35.468 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:35.468+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:35.745 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:35.743+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:36.453 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:36.453+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:36.492 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:36.491+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:36.766 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:36.765+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:37.491 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:37.490+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:37.515 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:37.514+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:37.757 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:37.756+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:38.457 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:38.456+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:38.508 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:38.508+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:38.760 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:38.759+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:39.441 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:39.440+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:39.522 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:39.521+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:39.800 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:39.799+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:40.476 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:40.476+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:40.557 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:40.557+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:40.772 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:40.772+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:41.432 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:41.431+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:41.553 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:41.552+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:41.748 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:41.747+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:42.392 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:42.392+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:42.515 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:42.515+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:42.706 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:42.705+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:43.415 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:43.414+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:43.533 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:43.533+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:43.729 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:43.728+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:44.434 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:44.434+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:44.491 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:44.490+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:44.745 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:44.745+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:45.452 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:45.453+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:45.503 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:45.503+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:45.787 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:45.787+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:46.459 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:46.459+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:46.481 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:46.482+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:46.760 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:46.761+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:47.430 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:47.431+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:47.514 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:47.515+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:47.797 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:47.797+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:48.399 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:48.400+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:48.481 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:48.481+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:48.781 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:48.782+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:49.375 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:49.375+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:49.499 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:49.499+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:49.810 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:49.810+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:50.355 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:50.356+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:50.465 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:50.465+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:50.788 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:50.788+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:51.313 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:51.313+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:51.417 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:51.417+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:51.831 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:51.831+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:52.331 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:52.332+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:52.440 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:52.440+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:52.800 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:52.800+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:53.309 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:53.309+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:53.466 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:53.466+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:53.770 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:53.771+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:54.317 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:54.317+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:54.431 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:54.432+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:54.799 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:54.799+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:55.327 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:55.327+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:55.464 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:55.465+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:55.755 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:55.755+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:56.335 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:56.335+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:56.457 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:56.457+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:56.804 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:56.805+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:57.320 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:57.320+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:57.467 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:57.467+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:57.768 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:57.768+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:58.342 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:58.341+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:58.498 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:58.497+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:58.772 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:58.771+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:59.338 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:48:59.337+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:59.511 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:48:59.511+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:48:59.746 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:48:59.746+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:00.324 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:00.324+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:00.487 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:00.487+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:00.698 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:00.698+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:01.349 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:01.348+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:01.473 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:01.472+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:01.703 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:01.703+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:02.307 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:02.306+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:02.513 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:02.512+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:02.734 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:02.733+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:03.271 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:03.270+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:03.559 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:03.558+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:03.695 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:03.694+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:04.254 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:04.254+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:04.521 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:04.521+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:04.715 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:04.714+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:05.266 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:05.266+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:05.484 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:05.483+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:05.695 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:05.695+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:06.268 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:06.267+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:06.518 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:06.517+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:06.690 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:06.689+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:07.221 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:07.220+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:07.555 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:07.554+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:07.729 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:07.728+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:08.187 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:08.186+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:08.508 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:08.507+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:08.701 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:08.700+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:09.163 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:09.162+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:09.459 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:09.458+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:09.672 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:09.671+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:10.149 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:10.148+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:10.466 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:10.466+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:10.719 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:10.718+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:11.201 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:11.198+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:11.462 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:11.461+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:11.768 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:11.768+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:12.249 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:12.249+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:12.442 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:12.441+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:12.809 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:12.808+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:13.219 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:13.219+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:13.491 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:13.490+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:13.847 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:13.846+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:14.257 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:14.257+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:14.488 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:14.488+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:14.895 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:14.894+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:15.222 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:15.221+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:15.462 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:15.461+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:15.933 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:15.932+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:16.177 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:16.177+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:16.476 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:16.475+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:16.963 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:16.962+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:17.189 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:17.188+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:17.442 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:17.441+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:17.984 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:17.983+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:18.198 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:18.197+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:18.492 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:18.491+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:19.012 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:19.011+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:19.241 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:19.240+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:19.463 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:19.462+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:19.993 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:19.993+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:20.275 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:20.275+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:20.454 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:20.453+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:20.965 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:20.965+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:21.265 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:21.264+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:21.460 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:21.460+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:22.015 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:22.015+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:22.248 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:22.247+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:22.422 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:22.422+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:23.012 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:23.012+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:23.206 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:23.206+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:23.397 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:23.396+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:24.024 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:24.023+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:24.247 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:24.247+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:24.375 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:24.374+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:24.996 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:24.995+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:25.260 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:25.259+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:25.371 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:25.370+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:25.953 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:25.952+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:26.307 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:26.306+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:26.405 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:26.404+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:26.916 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:26.916+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:27.260 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:27.259+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:27.385 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:27.383+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:27.913 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:27.913+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:28.310 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:28.309+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:28.433 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:28.432+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:28.931 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:28.932+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:29.313 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:29.313+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:29.393 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:29.393+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:29.953 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:29.953+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:30.323 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:30.323+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:30.419 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:30.420+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:30.904 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:30.904+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:31.339 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:31.339+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:31.441 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:31.441+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:31.915 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:31.915+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:32.311 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:32.311+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:32.427 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:32.428+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:32.895 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:32.896+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:33.272 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:33.272+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:33.405 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:33.405+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:33.944 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:33.944+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:34.306 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:34.306+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:34.366 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:34.366+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:34.904 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:34.905+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:35.319 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:35.320+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:35.325 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:35.325+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:35.903 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:35.903+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:36.294 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:36.294+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:36.348 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:36.348+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:36.909 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:36.909+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:37.316 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:37.316+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:37.345 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:37.345+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:37.941 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:37.941+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:38.317 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:38.317+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:38.387 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:38.388+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:38.979 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:38.980+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:39.355 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:39.355+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:39.373 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:39.373+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:39.956 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:39.956+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:40.314 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:40.314+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:40.330 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:40.330+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:41.002 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:41.002+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:41.319 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:41.319+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:41.331 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:41.331+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:41.960 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:41.960+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:42.353 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:42.352+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:42.379 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:42.378+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:42.996 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:42.995+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:43.310 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:43.309+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:43.405 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:43.405+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:43.983 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:43.982+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:44.262 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:44.261+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:44.383 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:44.382+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 14 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:44.944 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:44.943+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:45.299 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:45.299+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:45.413 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:45.412+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 13 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:45.952 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:45.951+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:46.350 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:46.349+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:46.461 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:46.460+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 13 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:46.918 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:46.917+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:47.323 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:47.322+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:47.424 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:47.423+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 13 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:47.919 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:47.918+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:48.345 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:48.344+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:48.453 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:48.452+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 13 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:48.894 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:48.893+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:49.377 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:49.377+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:49.476 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:49.475+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 13 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:49.853 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:49.852+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:50.355 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:50.355+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:50.478 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:50.477+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 13 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:50.813 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:50.813+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:51.389 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:51.389+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:51.469 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:51.469+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 13 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:51.831 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:51.831+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:52.357 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:52.356+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:52.463 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:52.462+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 13 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:52.812 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:52.811+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:53.335 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:53.334+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:53.480 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:53.479+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 13 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:53.837 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:53.837+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:54.298 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:54.297+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:54.432 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:54.431+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 13 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:54.837 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:54.836+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:55.254 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:55.253+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:55.465 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:55.464+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 13 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:55.810 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:55.809+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:56.253 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:56.252+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:56.500 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:56.498+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 13 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:56.763 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:56.762+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:57.295 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:57.295+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:57.518 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:57.518+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 13 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:57.805 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:57.804+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:58.342 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:58.341+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:58.485 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:58.484+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 13 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:58.786 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:58.786+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:59.308 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:49:59.307+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:59.446 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:49:59.445+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 13 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:49:59.810 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:49:59.810+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:00.284 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:00.283+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:00.401 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:00.400+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 13 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:00.793 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:00.792+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:01.311 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:01.311+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:01.401 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:01.400+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 13 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:01.764 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:01.763+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:02.317 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:02.316+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:02.449 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:02.446+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 13 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:02.807 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:02.806+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:03.307 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:03.306+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:03.488 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:03.487+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 13 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:03.769 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:03.769+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:04.326 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:04.325+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:04.526 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:04.525+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 13 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:04.783 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:04.782+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:05.278 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:05.277+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:05.557 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:05.556+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 13 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:05.774 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:05.773+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:06.291 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:06.290+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:06.587 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:06.586+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 13 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:06.782 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:06.781+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:07.311 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:07.310+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:07.542 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:07.541+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 13 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:07.810 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:07.809+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:08.267 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:08.267+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:08.538 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:08.537+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 13 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:08.843 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:08.842+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:09.288 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:09.287+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:09.491 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:09.491+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 13 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:09.885 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:09.884+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:10.267 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:10.266+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:10.537 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:10.537+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 13 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:10.922 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:10.921+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:11.314 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:11.313+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:11.513 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:11.512+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 13 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:11.968 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:11.968+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:12.352 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:12.351+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:12.557 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:12.558+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 13 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:12.943 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:12.942+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:13.311 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:13.310+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:13.604 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:13.603+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 13 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:13.988 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:13.988+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:14.285 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:14.285+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:14.603 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:14.603+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 13 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:14.994 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:14.995+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:15.289 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:15.289+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:15.636 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:15.637+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:16.017 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:16.018+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:16.321 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:16.321+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:16.657 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:16.657+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:16.997 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:16.998+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:17.302 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:17.302+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:17.621 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:17.622+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:18.043 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:18.043+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:18.327 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:18.328+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:18.657 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:18.657+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:18.994 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:18.995+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:19.312 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:19.312+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:19.695 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:19.696+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:20.012 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:20.012+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:20.295 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:20.295+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:20.734 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:20.734+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:21.029 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:21.029+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:21.335 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:21.335+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:21.749 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:21.750+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:22.033 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:22.033+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:22.345 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:22.346+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:22.741 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:22.741+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:23.017 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:23.017+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:23.375 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:23.376+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:23.717 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:23.718+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:23.986 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:23.987+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:24.407 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:24.407+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:24.696 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:24.697+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:25.003 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:25.004+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:25.452 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:25.452+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:25.672 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:25.672+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:25.968 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:25.969+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:26.469 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:26.469+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:26.693 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:26.693+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:26.949 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:26.948+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:27.519 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:27.518+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:27.709 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:27.708+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:27.927 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:27.926+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:28.473 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:28.473+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:28.681 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:28.680+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:28.954 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:28.953+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:29.447 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:29.446+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:29.712 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:29.711+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:29.912 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:29.911+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:30.403 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:30.402+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:30.757 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:30.756+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:30.901 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:30.900+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:31.380 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:31.379+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:31.802 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:31.801+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:31.937 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:31.936+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:32.334 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:32.333+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:32.788 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:32.787+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:32.962 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:32.961+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:33.285 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:33.284+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:33.819 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:33.818+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:33.946 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:33.945+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:34.317 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:34.317+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:34.801 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:34.801+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:34.973 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:34.973+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:35.358 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:35.357+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:35.806 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:35.806+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:35.984 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:35.983+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:36.374 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:36.373+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:36.830 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:36.829+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:37.016 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:37.015+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:37.414 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:37.413+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:37.787 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:37.786+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:38.059 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:38.058+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:38.386 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:38.385+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:38.762 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:38.762+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:39.035 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:39.034+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:39.345 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:39.344+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:39.806 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:39.805+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:40.077 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:40.077+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:40.311 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:40.311+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:40.808 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:40.808+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:41.067 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:41.067+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:41.358 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:41.357+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:41.850 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:41.850+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:42.034 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:42.033+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:42.316 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:42.315+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:42.856 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:42.855+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:42.997 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:42.997+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:43.305 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:43.305+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:43.905 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:43.904+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:44.005 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:44.004+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:44.298 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:44.297+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:44.890 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:44.889+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:45.001 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:45.000+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:45.251 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:45.250+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:45.896 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:45.895+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:46.018 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:46.018+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:46.282 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:46.281+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:46.869 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:46.868+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:47.067 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:47.066+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:47.283 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:47.283+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:47.821 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:47.820+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:48.045 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:48.044+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:48.292 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:48.291+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:48.811 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:48.810+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:49.039 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:49.038+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:49.265 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:49.264+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:49.777 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:49.776+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:50.040 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:50.039+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:50.233 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:50.232+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:50.800 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:50.800+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:51.029 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:51.028+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:51.189 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:51.189+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:51.821 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:51.821+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:52.008 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:52.007+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:52.150 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:52.150+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:52.835 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:52.834+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:53.013 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:53.013+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:53.133 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:53.133+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:53.825 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:53.824+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:54.023 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:54.022+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:54.121 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:54.120+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:54.831 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:54.830+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:54.974 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:54.974+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:55.108 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:55.107+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:55.861 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:55.861+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:56.016 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:56.015+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:56.064 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:56.063+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:56.840 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:56.840+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:57.042 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:57.042+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:57.049 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:57.048+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:57.831 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:57.831+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:58.050 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:58.050+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:58.055 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:58.055+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:58.817 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:58.817+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:59.009 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:59.009+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:59.098 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:50:59.098+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:59.769 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:50:59.770+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:50:59.985 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:50:59.985+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:00.082 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:00.083+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:00.808 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:00.808+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:00.993 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:00.993+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:01.125 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:01.125+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:01.811 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:01.811+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:02.035 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:02.035+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:02.111 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:02.111+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:02.841 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:02.841+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:03.028 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:03.028+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:03.065 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:03.065+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:03.829 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:03.829+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:04.010 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:04.010+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:04.073 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:04.073+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:04.799 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:04.799+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:04.975 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:04.975+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:05.106 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:05.106+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:05.844 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:05.844+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:05.968 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:05.969+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:06.063 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:06.063+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:06.836 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:06.836+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:07.005 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:07.005+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:07.065 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:07.065+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:07.792 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:07.792+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:08.019 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:08.019+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:08.090 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:08.091+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:08.755 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:08.755+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:08.983 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:08.983+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:09.056 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:09.056+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:09.793 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:09.793+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:09.996 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:09.997+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:10.049 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:10.050+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:10.806 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:10.806+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:11.039 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:11.038+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:11.046 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:11.046+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:11.824 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:11.823+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:12.010 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:12.009+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:12.059 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:12.058+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:12.813 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:12.812+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:13.018 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:13.018+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:13.078 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:13.077+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:13.780 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:13.779+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:14.036 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:14.036+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:14.060 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:14.059+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:14.829 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:14.828+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:15.039 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:15.039+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:15.059 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:15.058+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:15.820 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:15.820+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:16.030 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:16.029+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:16.091 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:16.090+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:16.789 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:16.788+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:17.048 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:17.047+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:17.050 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:17.049+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:17.814 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:17.813+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:18.058 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:18.058+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:18.086 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:18.085+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:18.831 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:18.831+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:19.025 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:19.024+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:19.082 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:19.082+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:19.845 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:19.844+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:20.002 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:20.001+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:20.092 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:20.092+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:20.828 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:20.827+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:21.043 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:21.043+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:21.089 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:21.086+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:21.813 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:21.813+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:22.006 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:22.006+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:22.076 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:22.075+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:22.804 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:22.804+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:22.974 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:22.974+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:23.046 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:23.045+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:23.756 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:23.755+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:23.952 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:23.952+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:24.089 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:24.088+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:24.780 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:24.780+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:24.946 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:24.945+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:25.118 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:25.117+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:25.800 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:25.799+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:25.984 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:25.983+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:26.121 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:26.120+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:26.823 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:26.820+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:27.013 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:27.012+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:27.131 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:27.129+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:27.844 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:27.844+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:28.025 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:28.024+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:28.163 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:28.162+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:28.881 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:28.880+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:28.988 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:28.987+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:29.124 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:29.123+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:29.843 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:29.843+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:30.037 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:30.036+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:30.124 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:30.123+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:30.855 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:30.854+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:31.009 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:31.008+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:31.109 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:31.108+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:31.870 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:31.869+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:31.993 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:31.990+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:32.141 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:32.140+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:32.893 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:32.893+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:32.986 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:32.985+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:33.169 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:33.167+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:33.860 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:33.859+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:33.938 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:33.938+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:34.145 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:34.144+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:34.903 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:34.902+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:34.983 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:34.982+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:35.111 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:35.110+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:35.906 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:35.905+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:36.005 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:36.005+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:36.157 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:36.156+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:36.924 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:36.923+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:36.998 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:36.997+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:37.166 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:37.166+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:37.955 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:37.955+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:37.964 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:37.963+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:38.200 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:38.200+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:38.941 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:38.940+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:38.986 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:38.985+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:39.163 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:39.163+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:39.968 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:39.968+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:39.989 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:39.989+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:40.189 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:40.188+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:40.956 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:40.955+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:41.009 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:41.009+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:41.199 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:41.199+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:41.946 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:41.947+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:42.043 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:42.042+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:42.196 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:42.193+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:42.942 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:42.942+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:43.014 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:43.014+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:43.197 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:43.197+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:43.962 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:43.962+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:43.985 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:43.985+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:44.181 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:44.181+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:44.962 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:44.962+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:45.024 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:45.024+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:45.206 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:45.207+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:45.946 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:45.946+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:46.026 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:46.027+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:46.220 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:46.221+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:46.925 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:46.926+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:47.017 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:47.017+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:47.190 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:47.190+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:47.926 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:47.926+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:48.045 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:48.045+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:48.154 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:48.154+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:48.956 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:48.956+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:49.052 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:49.052+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:49.167 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:49.168+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:49.990 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:49.990+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:50.010 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:50.010+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:50.184 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:50.184+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:50.983 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:50.983+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:51.038 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:51.038+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:51.162 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:51.163+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:51.977 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:51.977+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:52.021 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:52.022+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:52.119 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:52.119+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:52.971 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:52.971+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:53.049 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:53.049+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:53.125 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:53.125+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:53.994 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:53.995+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:54.045 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:54.046+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:54.112 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:54.113+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:54.979 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:54.979+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:55.065 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:55.065+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:55.129 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:55.129+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:55.972 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:55.971+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:56.088 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:56.087+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:56.111 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:56.110+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:56.940 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:56.939+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:57.040 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:57.040+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:57.078 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:57.077+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:57.953 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:57.952+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:58.079 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:58.078+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:58.096 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:58.095+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:58.943 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:58.942+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:59.122 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:51:59.121+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:59.125 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:51:59.124+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:51:59.970 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:51:59.969+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:00.115 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:00.115+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:00.143 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:00.142+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:00.977 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:00.976+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:01.128 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:01.127+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:01.142 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:01.141+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:01.987 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:01.986+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:02.121 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:02.120+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:02.157 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:02.156+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:02.961 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:02.960+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:03.094 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:03.093+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:03.194 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:03.193+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:03.943 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:03.942+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:04.131 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:04.131+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:04.224 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:04.224+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:04.960 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:04.959+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:05.119 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:05.118+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:05.265 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:05.264+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:05.979 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:05.979+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:06.128 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:06.127+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:06.237 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:06.236+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:06.970 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:06.970+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:07.130 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:07.129+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:07.203 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:07.203+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:07.946 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:07.945+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:08.109 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:08.108+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:08.206 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:08.206+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:08.976 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:08.975+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:09.093 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:09.092+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:09.201 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:09.200+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:10.012 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:10.011+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:10.092 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:10.091+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:10.174 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:10.173+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:10.987 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:10.986+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:11.068 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:11.067+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:11.215 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:11.215+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:12.007 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:12.006+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:12.038 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:12.038+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:12.235 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:12.235+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:13.011 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:13.010+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:13.016 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:13.015+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:13.249 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:13.248+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:14.014 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:14.013+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:14.032 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:14.031+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:14.207 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:14.207+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:15.009 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:15.008+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:15.071 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:15.070+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:15.161 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:15.161+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:16.045 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:16.044+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:16.101 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:16.101+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:16.161 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:16.160+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:17.076 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:17.075+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:17.091 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:17.091+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:17.122 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:17.121+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:18.070 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:18.069+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:18.094 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:18.093+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:18.112 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:18.111+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:19.052 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:19.051+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:19.099 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:19.098+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:19.120 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:19.119+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:20.059 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:20.058+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:20.095 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:20.094+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:20.104 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:20.104+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:21.019 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:21.018+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:21.072 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:21.071+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:21.094 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:21.094+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:21.977 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:21.976+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:22.082 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:22.081+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:22.096 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:22.095+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:22.971 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:22.970+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:23.055 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:23.055+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:23.070 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:23.069+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:23.952 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:23.952+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:24.033 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:24.032+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:24.044 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:24.043+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:24.986 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:24.985+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:25.019 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:25.018+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:25.084 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:25.083+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:25.987 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:25.986+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:25.999 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:25.999+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:26.103 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:26.103+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:26.962 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:26.962+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:27.013 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:27.013+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:27.104 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:27.104+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:27.957 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:27.957+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:27.988 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:27.989+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:28.128 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:28.129+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:28.914 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:28.915+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:28.962 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:28.963+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:29.162 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:29.162+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:29.902 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:29.902+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:29.958 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:29.958+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:30.159 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:30.159+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:30.913 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:30.914+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:30.943 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:30.944+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:31.148 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:31.148+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:31.947 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:31.947+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:31.993 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:31.993+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:32.172 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:32.172+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:32.957 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:32.958+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:33.004 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:33.004+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:33.209 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:33.209+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:33.962 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:33.962+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:33.975 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:33.975+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:34.161 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:34.161+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:34.944 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:34.944+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:34.999 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:34.999+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:35.190 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:35.190+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:35.912 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:35.912+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:35.974 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:35.975+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:36.166 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:36.166+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:36.899 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:36.899+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:36.939 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:36.939+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:37.216 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:37.216+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:37.902 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:37.902+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:37.937 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:37.937+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:38.221 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:38.221+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:38.856 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:38.855+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:38.974 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:38.974+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:39.211 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:39.211+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:39.825 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:39.825+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:39.973 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:39.972+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:40.193 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:40.192+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:40.798 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:40.798+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:41.001 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:41.000+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:41.168 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:41.167+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:41.775 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:41.775+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:41.963 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:41.962+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:42.121 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:42.121+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:42.799 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:42.799+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:42.967 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:42.967+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:43.115 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:43.114+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:43.828 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:43.827+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:43.997 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:43.996+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:44.082 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:44.081+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:44.801 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:44.800+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:44.981 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:44.981+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:45.051 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:45.050+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:45.842 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:45.841+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:45.962 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:45.962+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:46.085 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:46.084+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:46.828 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:46.827+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:46.982 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:46.982+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:47.102 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:47.102+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:47.779 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:47.778+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:47.990 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:47.989+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:48.114 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:48.113+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:48.731 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:48.730+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:48.978 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:48.977+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:49.094 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:49.094+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:49.776 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:49.775+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:49.962 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:49.961+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:50.057 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:50.056+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:50.736 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:50.735+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:50.940 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:50.939+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:51.072 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:51.071+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:51.779 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:51.778+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:51.983 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:51.982+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:52.119 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:52.118+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:52.777 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:52.776+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:53.019 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:53.018+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:53.146 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:53.145+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:53.764 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:53.764+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:54.025 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:54.024+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:54.192 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:54.191+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:54.801 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:54.801+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:54.998 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:54.998+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:55.167 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:55.166+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:55.829 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:55.828+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:56.043 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:56.042+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:56.198 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:56.198+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:56.875 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:56.874+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:57.066 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:57.066+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:57.210 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:57.210+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:57.870 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:57.869+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:58.081 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:58.080+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:58.221 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:58.220+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:58.892 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:58.891+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:59.129 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:52:59.128+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:59.208 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:52:59.207+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:52:59.915 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:52:59.914+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:00.156 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:00.155+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:00.219 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:00.219+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:00.916 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:00.916+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:01.152 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:01.151+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:01.254 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:01.254+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:01.889 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:01.889+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:02.118 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:02.117+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:02.229 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:02.228+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:02.909 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:02.908+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:03.132 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:03.132+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:03.217 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:03.217+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:03.878 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:03.877+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:04.090 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:04.089+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:04.196 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:04.195+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:04.908 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:04.907+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:05.136 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:05.135+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:05.185 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:05.184+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:05.958 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:05.957+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:06.139 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:06.138+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:06.140 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:06.139+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:06.911 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:06.911+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:07.099 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:07.098+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:07.168 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:07.168+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:07.927 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:07.927+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:08.137 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:08.136+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:08.195 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:08.194+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:08.933 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:08.931+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:09.149 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:09.148+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:09.218 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:09.218+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:09.911 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:09.910+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:10.171 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:10.171+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:10.224 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:10.223+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:10.875 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:10.874+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:11.190 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:11.190+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:11.207 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:11.206+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:11.916 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:11.916+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:12.169 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:12.169+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:12.222 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:12.222+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:12.947 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:12.947+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:13.196 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:13.196+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:13.248 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:13.249+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:13.968 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:13.969+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:14.174 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:14.175+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:14.218 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:14.219+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:14.945 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:14.945+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:15.186 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:15.186+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:15.221 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:15.221+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:15.947 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:15.947+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:16.142 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:16.142+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:16.258 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:16.259+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:16.960 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:16.960+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:17.109 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:17.109+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:17.298 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:17.298+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:17.930 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:17.930+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:18.133 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:18.133+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:18.283 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:18.283+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:18.972 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:18.973+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:19.130 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:19.130+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:19.236 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:19.237+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:19.968 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:19.968+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:20.131 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:20.131+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:20.257 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:20.257+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:20.984 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:20.984+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:21.155 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:21.156+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:21.264 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:21.265+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:21.950 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:21.950+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:22.131 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:22.131+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:22.260 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:22.261+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:22.901 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:22.901+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:23.173 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:23.173+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:23.264 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:23.264+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:23.900 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:23.900+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:24.182 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:24.182+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:24.303 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:24.302+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:24.916 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:24.915+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:25.146 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:25.145+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:25.323 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:25.322+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:25.894 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:25.893+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:26.110 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:26.110+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:26.294 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:26.293+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:26.856 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:26.855+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:27.092 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:27.091+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:27.336 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:27.336+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:27.849 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:27.848+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:28.114 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:28.113+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:28.298 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:28.298+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:28.898 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:28.898+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:29.070 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:29.069+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:29.293 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:29.292+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:29.929 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:29.928+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:30.052 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:30.052+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:30.338 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:30.338+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:30.927 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:30.926+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:31.021 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:31.021+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:31.376 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:31.375+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:31.920 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:31.920+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:32.035 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:32.035+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:32.362 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:32.361+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:32.910 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:32.909+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:33.038 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:33.038+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:33.334 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:33.333+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:33.917 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:33.917+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:34.064 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:34.064+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:34.358 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:34.357+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:34.967 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:34.966+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:35.070 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:35.069+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:35.344 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:35.343+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:35.931 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:35.930+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:36.112 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:36.112+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:36.319 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:36.319+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:36.918 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:36.917+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:37.109 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:37.108+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:37.309 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:37.308+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:37.873 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:37.872+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:38.099 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:38.098+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:38.358 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:38.357+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:38.847 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:38.846+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:39.094 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:39.093+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:39.374 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:39.373+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:39.874 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:39.873+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:40.139 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:40.138+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:40.381 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:40.380+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:40.840 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:40.839+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:41.130 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:41.129+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:41.413 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:41.412+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:41.838 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:41.837+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:42.149 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:42.149+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:42.418 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:42.417+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:42.873 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:42.872+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:43.178 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:43.177+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:43.440 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:43.440+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:43.843 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:43.843+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:44.168 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:44.167+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:44.419 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:44.418+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:44.830 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:44.829+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:45.203 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:45.203+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:45.438 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:45.437+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:45.878 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:45.877+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:46.203 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:46.203+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:46.441 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:46.440+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:46.835 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:46.834+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:47.241 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:47.241+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:47.433 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:47.432+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:47.860 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:47.859+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:48.205 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:48.204+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:48.481 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:48.480+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:48.860 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:48.860+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:49.174 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:49.173+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:49.437 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:49.437+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:49.870 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:49.869+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:50.127 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:50.126+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:50.394 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:50.393+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:50.918 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:50.917+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:51.143 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:51.142+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:51.368 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:51.367+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:51.880 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:51.879+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:52.133 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:52.132+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:52.346 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:52.345+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:52.830 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:52.830+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:53.097 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:53.097+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:53.323 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:53.322+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:53.801 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:53.801+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:54.062 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:54.061+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:54.322 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:54.321+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:54.837 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:54.836+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:55.038 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:55.038+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:55.292 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:55.291+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:55.794 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:55.795+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:56.089 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:56.089+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:56.248 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:56.248+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:56.785 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:56.785+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:57.074 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:57.074+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:57.287 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:57.287+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:57.774 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:57.775+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:58.111 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:58.112+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:58.274 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:58.274+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:58.803 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:58.803+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:59.131 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:53:59.131+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:59.291 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:53:59.291+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:53:59.778 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:53:59.779+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:00.146 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:00.147+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:00.340 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:00.340+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:00.758 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:00.758+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:01.156 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:01.156+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:01.387 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:01.387+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:01.769 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:01.769+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:02.114 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:02.114+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:02.382 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:02.382+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:02.756 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:02.756+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:03.118 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:03.119+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:03.336 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:03.336+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:03.707 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:03.708+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:04.118 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:04.119+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:04.334 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:04.335+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:04.747 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:04.747+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:05.150 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:05.151+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:05.290 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:05.291+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:05.748 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:05.748+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:06.143 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:06.143+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:06.315 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:06.315+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:06.763 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:06.764+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:07.123 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:07.123+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:07.347 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:07.348+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:07.728 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:07.728+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:08.173 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:08.173+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:08.330 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:08.330+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:08.697 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:08.696+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:09.180 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:09.179+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:09.335 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:09.334+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:09.672 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:09.671+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:10.196 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:10.196+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:10.349 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:10.348+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:10.644 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:10.644+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:11.226 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:11.225+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:11.345 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:11.344+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:11.663 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:11.662+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:12.213 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:12.212+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:12.316 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:12.316+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:12.619 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:12.618+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:13.252 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:13.251+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:13.364 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:13.364+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:13.579 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:13.579+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:14.268 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:14.267+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:14.326 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:14.325+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:14.617 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:14.616+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:15.317 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:15.316+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:15.331 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:15.330+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:15.662 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:15.662+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:16.334 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:16.334+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:16.346 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:16.346+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:16.632 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:16.631+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:17.309 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:17.308+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:17.353 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:17.352+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:17.673 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:17.673+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:18.329 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:18.329+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:18.337 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:18.337+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:18.710 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:18.709+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:19.322 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:19.321+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:19.372 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:19.371+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:19.708 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:19.707+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:20.345 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:20.345+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:20.363 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:20.362+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:20.712 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:20.711+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:21.315 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:21.314+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:21.323 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:21.322+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:21.666 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:21.665+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:22.285 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:22.284+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:22.365 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:22.364+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:22.631 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:22.631+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:23.246 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:23.245+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:23.350 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:23.349+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:23.588 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:23.587+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:24.216 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:24.215+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:24.302 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:24.302+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:24.605 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:24.605+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:25.199 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:25.198+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:25.262 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:25.262+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:25.582 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:25.582+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:26.215 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:26.214+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:26.292 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:26.291+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:26.597 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:26.596+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:27.246 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:27.245+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:27.256 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:27.255+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:27.597 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:27.597+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:28.229 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:28.229+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:28.246 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:28.245+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:28.589 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:28.588+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:29.211 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:29.211+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:29.236 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:29.235+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:29.630 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:29.630+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:30.192 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:30.192+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:30.204 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:30.204+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:30.643 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:30.643+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:31.162 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:31.161+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:31.165 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:31.165+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:31.649 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:31.648+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:32.117 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:32.116+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:32.198 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:32.198+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:32.639 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:32.638+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:33.131 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:33.131+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:33.245 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:33.244+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:33.686 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:33.685+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:34.109 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:34.108+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:34.239 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:34.238+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:34.672 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:34.671+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:35.099 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:35.098+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:35.282 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:35.281+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:35.691 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:35.690+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:36.064 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:36.064+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:36.279 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:36.279+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:36.680 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:36.679+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:37.109 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:37.109+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:37.322 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:37.321+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:37.710 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:37.709+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:38.132 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:38.131+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:38.320 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:38.319+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:38.727 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:38.726+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:39.087 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:39.086+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:39.321 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:39.320+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:39.750 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:39.750+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:40.127 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:40.127+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:40.275 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:40.276+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:40.769 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:40.769+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:41.082 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:41.083+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:41.309 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:41.309+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:41.756 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:41.756+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:42.078 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:42.078+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:42.304 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:42.305+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:42.767 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:42.767+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:43.057 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:43.057+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:43.283 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:43.284+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:43.814 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:43.815+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:44.079 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:44.079+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:44.304 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:44.305+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:44.798 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:44.798+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:45.058 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:45.058+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:45.291 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:45.291+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:45.806 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:45.806+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:46.099 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:46.099+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:46.265 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:46.265+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:46.766 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:46.766+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:47.139 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:47.139+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:47.222 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:47.223+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:47.758 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:47.759+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:48.141 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:48.141+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:48.261 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:48.261+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:48.803 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:48.804+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:49.132 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:49.133+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:49.252 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:49.253+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:49.808 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:49.809+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:50.133 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:50.133+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:50.256 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:50.257+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:50.818 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:50.819+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:51.168 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:51.169+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:51.301 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:51.302+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:51.814 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:51.814+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:52.141 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:52.141+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:52.342 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:52.342+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:52.808 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:52.808+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:53.130 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:53.129+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:53.324 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:53.324+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:53.780 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:53.780+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:54.123 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:54.122+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:54.293 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:54.292+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:54.776 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:54.775+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:55.144 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:55.143+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:55.244 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:55.244+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:55.742 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:55.741+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:56.165 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:56.164+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:56.232 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:56.231+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:56.702 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:56.701+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:57.136 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:57.136+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:57.192 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:57.192+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:57.705 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:57.704+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:58.154 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:58.153+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:58.167 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:58.166+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:58.704 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:58.703+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:59.132 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:54:59.131+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:59.204 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:54:59.204+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:54:59.658 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:54:59.657+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:00.174 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:00.173+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:00.238 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:00.237+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:00.671 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:00.670+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:01.173 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:01.172+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:01.284 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:01.284+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:01.691 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:01.690+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:02.212 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:02.211+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:02.328 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:02.327+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:02.682 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:02.681+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:03.237 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:03.237+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:03.335 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:03.334+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:03.682 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:03.681+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:04.235 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:04.234+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:04.329 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:04.328+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:04.728 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:04.728+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:05.194 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:05.193+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:05.338 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:05.337+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:05.711 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:05.710+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:06.164 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:06.164+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:06.295 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:06.294+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:06.733 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:06.732+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:07.199 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:07.199+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:07.345 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:07.344+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:07.717 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:07.713+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:08.212 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:08.211+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:08.367 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:08.366+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:08.757 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:08.757+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:09.240 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:09.239+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:09.412 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:09.412+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:09.763 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:09.762+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:10.241 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:10.240+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:10.408 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:10.407+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:10.756 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:10.755+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:11.236 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:11.234+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:11.373 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:11.372+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:11.786 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:11.785+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:12.286 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:12.285+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:12.394 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:12.394+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:12.798 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:12.797+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:13.333 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:13.333+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:13.403 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:13.402+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:13.832 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:13.831+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:14.368 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:14.353+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:14.431 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:14.430+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:14.859 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:14.858+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:15.324 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:15.323+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:15.400 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:15.399+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:15.898 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:15.897+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:16.304 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:16.303+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:16.412 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:16.411+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:16.948 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:16.947+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:17.294 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:17.294+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:17.406 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:17.405+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:17.925 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:17.924+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:18.255 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:18.254+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:18.438 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:18.438+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:18.926 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:18.926+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:19.276 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:19.275+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:19.472 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:19.471+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:19.891 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:19.890+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:20.239 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:20.238+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:20.430 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:20.430+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:20.898 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:20.897+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:21.258 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:21.257+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:21.480 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:21.479+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:21.904 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:21.904+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:22.258 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:22.257+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:22.444 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:22.443+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:22.869 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:22.869+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:23.239 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:23.239+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:23.397 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:23.396+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:23.858 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:23.858+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:24.263 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:24.263+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:24.358 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:24.359+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:24.883 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:24.883+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:25.238 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:25.239+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:25.408 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:25.408+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:25.911 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:25.912+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:26.208 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:26.209+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:26.427 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:26.428+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:26.908 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:26.908+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:27.248 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:27.248+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:27.393 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:27.393+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:27.901 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:27.901+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:28.298 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:28.298+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:28.365 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:28.365+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:28.942 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:28.942+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:29.324 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:29.324+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:29.400 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:29.401+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:29.982 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:29.982+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:30.285 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:30.285+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:30.445 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:30.446+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:30.946 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:30.946+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:31.251 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:31.251+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:31.473 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:31.473+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:31.932 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:31.932+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:32.238 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:32.238+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:32.514 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:32.514+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:32.890 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:32.890+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:33.264 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:33.264+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:33.517 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:33.517+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:33.870 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:33.871+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:34.249 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:34.249+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:34.541 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:34.542+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:34.867 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:34.868+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:35.211 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:35.212+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:35.585 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:35.585+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:35.893 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:35.893+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:36.196 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:36.196+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:36.573 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:36.574+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:36.920 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:36.920+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:37.192 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:37.192+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:37.602 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:37.601+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:37.880 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:37.879+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:38.205 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:38.204+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:38.624 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:38.624+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:38.854 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:38.853+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:39.225 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:39.225+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:39.601 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:39.600+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:39.859 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:39.858+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:40.216 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:40.216+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:40.643 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:40.642+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:40.885 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:40.885+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:41.246 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:41.245+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:41.651 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:41.650+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:41.897 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:41.896+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:42.271 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:42.270+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:42.653 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:42.652+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:42.880 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:42.879+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:43.288 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:43.287+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:43.642 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:43.641+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:43.834 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:43.834+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:44.299 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:44.299+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:44.635 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:44.634+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:44.791 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:44.790+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:45.343 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:45.342+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:45.643 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:45.642+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:45.757 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:45.756+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:46.356 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:46.356+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:46.680 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:46.680+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:46.736 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:46.735+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:47.311 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:47.310+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:47.653 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:47.653+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:47.711 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:47.710+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:48.276 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:48.276+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:48.674 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:48.674+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:48.701 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:48.700+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:49.286 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:49.285+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:49.638 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:49.637+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:49.686 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:49.685+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:50.270 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:50.270+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:50.672 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:50.672+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:50.736 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:50.735+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:51.293 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:51.292+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:51.665 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:51.664+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:51.694 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:51.694+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:52.305 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:52.304+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:52.697 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:52.696+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:52.727 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:52.726+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:53.332 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:53.331+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:53.728 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:53.727+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:53.766 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:53.765+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:54.305 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:54.304+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:54.761 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:54.761+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:54.762 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:54.762+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:55.309 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:55.308+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:55.722 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:55.722+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:55.762 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:55.761+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:56.310 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:56.310+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:56.684 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:56.684+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:56.778 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:56.777+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:57.342 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:57.342+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:57.673 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:57.672+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:57.752 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:57.751+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:58.312 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:58.311+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:58.659 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:58.659+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:58.716 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:58.715+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:59.281 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:55:59.281+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:59.659 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:55:59.658+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:55:59.724 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:55:59.723+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:00.299 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:00.298+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:00.696 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:00.695+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:00.721 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:00.720+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:01.306 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:01.305+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:01.673 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:01.672+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:01.699 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:01.698+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:02.323 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:02.323+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:02.638 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:02.638+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:02.687 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:02.686+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:03.333 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:03.333+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:03.672 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:03.671+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:03.732 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:03.731+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:04.360 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:04.360+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:04.623 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:04.622+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:04.689 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:04.688+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:05.408 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:05.407+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:05.634 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:05.633+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:05.668 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:05.667+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:06.421 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:06.420+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:06.605 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:06.604+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:06.667 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:06.667+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:07.411 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:07.410+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:07.623 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:07.622+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:07.642 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:07.643+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:08.453 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:08.452+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:08.597 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:08.596+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:08.615 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:08.615+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:09.139 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T19:56:09.139+0000 7fccf4bd9700 0 -- 172.21.15.87:0/3193322219 >> [v2:172.21.15.87:6816/114598,v1:172.21.15.87:6817/114598] conn(0x7fccd80db130 msgr2=0x7fccd80e71c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-24T19:56:09.421 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:09.421+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:09.568 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:09.569+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:09.617 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:09.618+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:10.465 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:10.465+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:10.610 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:10.610+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:10.615 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:10.615+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:11.463 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:11.463+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:11.611 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:11.611+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:11.626 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:11.627+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:12.499 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:12.499+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:12.578 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:12.579+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:12.659 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:12.659+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:13.545 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:13.546+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:13.546 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:13.547+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:13.658 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:13.659+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:14.561 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:14.561+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:14.575 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:14.575+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:14.624 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:14.623+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:15.551 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:15.551+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:15.582 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:15.583+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:15.661 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:15.661+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:16.579 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:16.580+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:16.615 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:16.615+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:16.645 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:16.645+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:17.532 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:17.532+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:17.602 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:17.602+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:17.650 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:17.650+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:18.554 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:18.554+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:18.628 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:18.628+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:18.638 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:18.638+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:19.529 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:19.527+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:19.629 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:19.629+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:19.630 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:19.631+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:20.565 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:20.566+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:20.608 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:20.608+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:20.667 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:20.667+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:21.522 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:21.522+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:21.566 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:21.567+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:21.715 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:21.715+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:22.551 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:22.551+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:22.599 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:22.598+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:22.708 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:22.708+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:23.551 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:23.550+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:23.602 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:23.601+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:23.744 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:23.743+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:24.507 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:24.506+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:24.589 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:24.589+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:24.710 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:24.709+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:25.511 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:25.511+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:25.613 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:25.612+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:25.724 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:25.723+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:26.539 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:26.538+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:26.585 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:26.585+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:26.742 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:26.742+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:27.558 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:27.557+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:27.568 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:27.567+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:27.752 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:27.751+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:28.560 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:28.559+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:28.565 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:28.564+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:28.744 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:28.744+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:29.512 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:29.512+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:29.547 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:29.546+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:29.697 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:29.697+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:30.505 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:30.504+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:30.562 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:30.561+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:30.707 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:30.706+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:31.491 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:31.490+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:31.533 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:31.533+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:31.663 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:31.663+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:32.541 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:32.541+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:32.542 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:32.541+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:32.705 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:32.705+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:33.500 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:33.499+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:33.511 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:33.510+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:33.662 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:33.662+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:34.473 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:34.472+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:34.532 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:34.531+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:34.657 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:34.657+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:35.453 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:35.452+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:35.535 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:35.534+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:35.704 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:35.703+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:36.434 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:36.433+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:36.545 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:36.545+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:36.692 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:36.691+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:37.472 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:37.472+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:37.519 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:37.518+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:37.705 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:37.704+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:38.427 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:38.426+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:38.545 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:38.544+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:38.707 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:38.706+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:39.467 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:39.466+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:39.511 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:39.510+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:39.715 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:39.714+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:40.442 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:40.441+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:40.470 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:40.469+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:40.687 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:40.686+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:41.456 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:41.455+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:41.461 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:41.461+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:41.668 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:41.667+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:42.483 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:42.482+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:42.494 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:42.493+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:42.629 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:42.629+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:43.450 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:43.449+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:43.475 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:43.474+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:43.680 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:43.679+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:44.462 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:44.461+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:44.515 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:44.514+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:44.719 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:44.719+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:45.474 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:45.473+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:45.490 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:45.490+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:45.693 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:45.692+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:46.463 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:46.462+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:46.522 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:46.521+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:46.656 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:46.655+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:47.431 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:47.430+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:47.537 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:47.536+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:47.687 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:47.686+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:48.388 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:48.387+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:48.489 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:48.488+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:48.702 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:48.701+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:49.385 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:49.384+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:49.468 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:49.468+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:49.736 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:49.735+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:50.338 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:50.337+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:50.480 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:50.479+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:50.745 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:50.744+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:51.314 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:51.314+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:51.515 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:51.514+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:51.696 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:51.695+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:52.323 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:52.323+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:52.475 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:52.474+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:52.736 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:52.736+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:53.373 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:53.373+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:53.517 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:53.517+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:53.691 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:53.692+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:54.346 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:54.346+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:54.475 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:54.475+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:54.672 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:54.672+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:55.371 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:55.371+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:55.510 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:55.510+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:55.678 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:55.678+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:56.393 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:56.393+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:56.497 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:56.497+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:56.700 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:56.700+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:57.385 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:57.386+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:57.498 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:57.498+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:57.745 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:57.746+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:58.344 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:58.345+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:58.516 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:58.516+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:58.789 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:58.789+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:59.345 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:56:59.345+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:59.543 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:56:59.543+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:56:59.821 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:56:59.821+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:00.393 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:00.393+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:00.494 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:00.494+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:00.828 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:00.828+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:01.389 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:01.389+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:01.522 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:01.522+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:01.793 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:01.794+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:02.399 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:02.399+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:02.502 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:02.502+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:02.780 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:02.780+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:03.417 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:03.417+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:03.521 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:03.521+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:03.748 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:03.748+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:04.393 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:04.393+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:04.552 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:04.553+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:04.782 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:04.782+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:05.350 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:05.351+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:05.512 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:05.512+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:05.742 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:05.742+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:06.337 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:06.336+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:06.526 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:06.525+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:06.753 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:06.753+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:07.304 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:07.303+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:07.566 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:07.565+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:07.758 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:07.758+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:08.289 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:08.288+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:08.596 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:08.595+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:08.709 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:08.709+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:09.304 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:09.304+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:09.611 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:09.611+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:09.680 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:09.679+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:10.287 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:10.287+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:10.638 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:10.637+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:10.650 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:10.650+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:11.245 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:11.244+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:11.633 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:11.633+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:11.637 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:11.636+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:12.223 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:12.223+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:12.635 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:12.635+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:12.683 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:12.683+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:13.175 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:13.175+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:13.610 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:13.609+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:13.665 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:13.665+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:14.218 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:14.217+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:14.638 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:14.637+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:14.645 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:14.645+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:15.221 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:15.220+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:15.601 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:15.600+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:15.633 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:15.632+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:16.256 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:16.255+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:16.588 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:16.587+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:16.644 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:16.643+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:17.268 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:17.267+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:17.541 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:17.541+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:17.660 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:17.659+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:18.264 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:18.263+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:18.561 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:18.561+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:18.635 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:18.634+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:19.264 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:19.264+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:19.513 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:19.512+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:19.622 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:19.621+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:20.258 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:20.257+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:20.494 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:20.493+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:20.653 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:20.652+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:21.222 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:21.221+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:21.500 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:21.499+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:21.672 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:21.671+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:22.257 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:22.256+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:22.456 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:22.455+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:22.626 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:22.625+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:23.227 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:23.227+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:23.473 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:23.472+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:23.629 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:23.628+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:24.236 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:24.235+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:24.437 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:24.437+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:24.672 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:24.672+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:25.228 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:25.227+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:25.447 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:25.446+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:25.641 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:25.640+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:26.208 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:26.208+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:26.446 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:26.445+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:26.651 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:26.650+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:27.180 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:27.179+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:27.426 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:27.425+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:27.671 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:27.670+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:28.188 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:28.187+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:28.456 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:28.455+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:28.684 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:28.683+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:29.210 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:29.210+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:29.449 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:29.448+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:29.711 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:29.710+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:30.201 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:30.200+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:30.439 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:30.438+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:30.741 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:30.741+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:31.204 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:31.204+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:31.455 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:31.454+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:31.716 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:31.715+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:32.168 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:32.168+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:32.467 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:32.467+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:32.762 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:32.761+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:33.137 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:33.136+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:33.458 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:33.457+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:33.771 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:33.771+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:34.126 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:34.126+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:34.465 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:34.465+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:34.749 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:34.748+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:35.152 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:35.152+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:35.417 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:35.416+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:35.789 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:35.789+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:36.192 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:36.191+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:36.396 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:36.396+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:36.788 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:36.787+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:37.231 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:37.231+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:37.437 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:37.437+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:37.769 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:37.769+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:38.192 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:38.193+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:38.420 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:38.420+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:38.746 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:38.746+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:39.166 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:39.166+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:39.402 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:39.402+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:39.752 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:39.751+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:40.140 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:40.140+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:40.363 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:40.363+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:40.761 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:40.761+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:41.155 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:41.155+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:41.349 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:41.349+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:41.757 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:41.757+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:42.186 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:42.186+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:42.367 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:42.367+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:42.715 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:42.715+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:43.202 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:43.202+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:43.380 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:43.380+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:43.709 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:43.710+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:44.199 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:44.199+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:44.353 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:44.353+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:44.756 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:44.756+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:45.248 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:45.248+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:45.363 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:45.364+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:45.766 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:45.766+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:46.215 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:46.216+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:46.375 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:46.376+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:46.752 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:46.752+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:47.200 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:47.200+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:47.335 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:47.335+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:47.712 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:47.712+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:48.189 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:48.189+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:48.374 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:48.374+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:48.678 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:48.678+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:49.201 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:49.201+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:49.358 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:49.358+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:49.676 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:49.676+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:50.157 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:50.157+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:50.393 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:50.393+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:50.697 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:50.697+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:51.206 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:51.206+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:51.419 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:51.418+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:51.661 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:51.660+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:52.216 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:52.216+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:52.424 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:52.423+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:52.644 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:52.643+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:53.244 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:53.243+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:53.420 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:53.419+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:53.658 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:53.657+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:54.208 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:54.208+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:54.431 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:54.430+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:54.658 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:54.657+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:55.228 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:55.227+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:55.434 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:55.433+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:55.636 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:55.636+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:56.276 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:56.275+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:56.407 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:56.406+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:56.603 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:56.603+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:57.322 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:57.321+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:57.383 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:57.382+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:57.604 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:57.603+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:58.357 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:58.356+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:58.400 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:58.399+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:58.595 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:58.594+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:59.356 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:57:59.356+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:59.404 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:57:59.403+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:57:59.633 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:57:59.633+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:00.311 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:00.311+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:00.437 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:00.437+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:00.606 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:00.605+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:01.275 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:01.275+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:01.439 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:01.438+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:01.649 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:01.648+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:02.254 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:02.254+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:02.481 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:02.480+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:02.607 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:02.606+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:03.225 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:03.224+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:03.480 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:03.479+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:03.599 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:03.598+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:04.269 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:04.267+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:04.481 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:04.480+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:04.590 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:04.589+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:05.281 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:05.280+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:05.527 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:05.526+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:05.624 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:05.623+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:06.294 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:06.294+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:06.534 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:06.533+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:06.649 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:06.648+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:07.311 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:07.310+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:07.506 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:07.505+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:07.636 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:07.635+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:08.314 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:08.313+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:08.523 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:08.522+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:08.643 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:08.642+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:09.289 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:09.288+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:09.564 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:09.564+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:09.688 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:09.688+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:10.336 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:10.336+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:10.549 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:10.548+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:10.708 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:10.708+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:11.364 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:11.363+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:11.540 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:11.539+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:11.746 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:11.746+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:12.322 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:12.321+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:12.522 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:12.522+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:12.760 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:12.759+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:13.336 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:13.335+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:13.510 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:13.509+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:13.782 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:13.781+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:14.327 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:14.326+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:14.512 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:14.512+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:14.772 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:14.771+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:15.323 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:15.322+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:15.520 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:15.519+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:15.743 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:15.743+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:16.320 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:16.319+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:16.487 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:16.487+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:16.726 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:16.725+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:17.338 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:17.335+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:17.462 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:17.462+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:17.732 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:17.731+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:18.369 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:18.369+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:18.455 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:18.454+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:18.715 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:18.714+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:19.358 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:19.357+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:19.426 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:19.426+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:19.752 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:19.752+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:20.397 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:20.397+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:20.432 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:20.432+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:20.745 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:20.744+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:21.402 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:21.401+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:21.444 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:21.444+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:21.752 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:21.752+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:22.384 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:22.384+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:22.420 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:22.420+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:22.797 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:22.797+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:23.346 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:23.346+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:23.423 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:23.423+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:23.825 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:23.825+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:24.332 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:24.332+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:24.459 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:24.460+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:24.870 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:24.870+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:25.337 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:25.338+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:25.466 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:25.467+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:25.875 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:25.875+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:26.292 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:26.292+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:26.494 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:26.494+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:26.859 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:26.859+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:27.306 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:27.307+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:27.485 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:27.485+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:27.856 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:27.856+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:28.280 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:28.280+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:28.452 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:28.452+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:28.862 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:28.862+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:29.298 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:29.298+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:29.497 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:29.497+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:29.908 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:29.909+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:30.335 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:30.336+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:30.535 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:30.536+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:30.880 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:30.880+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:31.344 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:31.344+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:31.566 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:31.567+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:31.886 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:31.886+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:32.321 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:32.321+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:32.568 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:32.568+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:32.900 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:32.901+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:33.346 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:33.347+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:33.525 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:33.525+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:33.923 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:33.924+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:34.369 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:34.369+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:34.493 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:34.493+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:34.928 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:34.927+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:35.337 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:35.336+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:35.456 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:35.455+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:35.957 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:35.957+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:36.309 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:36.308+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:36.470 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:36.469+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:36.982 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:36.981+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:37.321 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:37.320+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:37.436 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:37.435+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:37.973 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:37.972+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:38.327 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:38.326+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:38.456 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:38.456+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:38.930 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:38.930+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:39.311 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:39.311+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:39.450 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:39.449+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:39.920 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:39.918+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:40.313 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:40.312+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:40.498 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:40.497+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:40.908 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:40.907+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:41.322 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:41.321+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:41.503 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:41.502+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:41.859 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:41.858+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:42.338 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:42.337+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:42.536 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:42.535+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:42.894 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:42.893+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:43.362 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:43.362+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:43.522 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:43.521+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:43.945 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:43.944+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:44.412 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:44.411+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:44.543 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:44.542+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:44.987 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:44.986+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:45.402 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:45.401+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:45.534 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:45.534+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:46.032 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:46.031+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:46.361 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:46.360+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:46.524 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:46.523+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:46.996 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:46.996+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:47.339 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:47.338+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:47.569 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:47.568+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:47.990 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:47.989+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:48.301 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:48.301+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:48.579 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:48.578+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:48.951 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:48.951+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:49.315 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:49.315+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:49.600 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:49.599+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:50.000 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:50.000+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:50.315 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:50.315+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:50.573 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:50.572+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:50.976 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:50.976+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:51.346 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:51.345+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:51.597 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:51.597+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:52.010 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:52.009+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:52.314 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:52.313+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:52.575 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:52.575+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:53.003 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:53.002+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:53.281 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:53.278+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:53.550 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:53.549+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:54.005 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:54.004+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:54.265 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:54.264+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:54.532 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:54.532+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:54.981 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:54.980+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:55.270 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:55.270+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:55.506 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:55.505+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:55.989 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:55.989+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:56.232 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:56.231+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:56.473 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:56.473+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:57.012 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:57.011+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:57.252 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:57.251+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:57.464 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:57.463+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:58.012 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:58.011+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:58.235 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:58.235+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:58.428 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:58.427+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:59.055 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:58:59.055+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:59.245 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:58:59.245+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:58:59.405 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:58:59.404+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:00.047 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:00.046+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:00.268 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:00.268+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:00.450 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:00.449+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:01.009 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:01.009+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:01.304 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:01.303+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:01.468 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:01.467+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:02.010 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:02.009+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:02.284 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:02.283+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:02.515 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:02.515+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:02.963 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:02.962+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:03.251 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:03.250+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:03.509 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:03.508+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:04.008 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:04.008+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:04.259 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:04.258+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:04.533 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:04.532+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:05.010 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:05.009+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:05.217 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:05.217+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:05.577 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:05.576+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:06.042 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:06.042+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:06.196 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:06.195+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:06.549 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:06.549+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:07.037 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:07.037+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:07.176 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:07.176+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:07.537 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:07.538+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:08.045 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:08.045+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:08.221 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:08.221+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:08.529 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:08.529+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:09.011 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:09.012+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:09.188 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:09.188+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:09.527 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:09.527+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:10.033 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:10.034+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:10.227 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:10.228+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:10.564 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:10.564+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:11.013 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:11.014+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:11.189 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:11.189+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:11.516 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:11.516+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:12.004 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:12.004+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:12.156 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:12.156+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:12.498 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:12.498+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:13.032 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:13.032+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:13.109 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:13.110+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:13.479 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:13.479+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:14.041 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:14.042+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:14.077 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:14.077+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:14.526 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:14.526+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:15.015 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:15.015+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:15.082 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:15.083+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:15.550 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:15.550+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:16.009 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:16.010+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:16.100 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:16.100+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:16.599 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:16.599+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:16.977 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:16.977+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:17.069 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:17.069+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:17.611 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:17.612+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:18.004 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:18.005+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:18.036 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:18.036+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:18.636 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:18.636+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:19.038 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:19.038+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:19.080 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:19.080+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:19.668 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:19.667+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:20.032 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:20.031+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:20.048 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:20.047+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:20.672 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:20.671+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:20.988 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:20.988+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:21.087 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:21.086+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:21.633 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:21.632+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:22.014 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:22.013+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:22.091 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:22.091+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:22.602 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:22.601+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:22.978 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:22.977+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:23.055 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:23.055+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:23.574 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:23.573+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:24.023 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:24.023+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:24.061 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:24.060+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:24.604 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:24.604+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:25.018 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:25.017+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:25.033 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:25.032+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:25.643 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:25.642+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:26.011 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:26.010+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:26.031 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:26.031+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:26.637 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:26.636+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:26.966 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:26.965+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:27.025 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:27.024+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:27.660 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:27.659+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:27.935 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:27.934+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:28.057 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:28.056+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:28.674 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:28.673+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:28.979 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:28.978+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:29.028 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:29.027+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:29.655 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:29.654+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:29.998 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:29.998+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:30.004 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:30.003+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:30.625 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:30.624+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:30.951 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:30.951+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:30.954 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:30.954+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:31.589 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:31.589+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:31.932 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:31.932+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:31.944 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:31.943+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:32.622 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:32.621+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:32.927 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:32.927+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:32.980 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:32.979+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:33.620 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:33.620+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:33.955 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:33.954+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:34.021 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:34.020+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:34.595 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:34.594+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:34.910 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:34.909+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:35.029 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:35.028+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:35.573 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:35.572+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:35.866 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:35.865+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:36.017 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:36.016+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:36.596 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:36.595+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:36.817 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:36.816+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:36.987 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:36.986+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:37.635 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:37.634+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:37.798 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:37.797+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:37.944 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:37.944+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:38.605 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:38.605+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:38.770 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:38.769+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:38.978 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:38.977+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:39.645 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:39.644+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:39.819 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:39.818+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:40.026 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:40.025+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:40.660 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:40.659+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:40.799 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:40.799+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:41.074 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:41.073+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:41.650 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:41.649+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:41.765 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:41.764+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:42.087 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:42.086+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:42.688 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:42.687+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:42.725 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:42.725+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:43.105 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:43.105+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:43.640 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:43.639+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:43.725 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:43.724+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:44.126 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:44.125+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:44.646 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:44.645+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:44.765 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:44.765+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:45.145 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:45.144+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:45.637 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:45.637+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:45.736 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:45.736+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:46.138 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:46.137+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:46.670 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:46.669+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:46.751 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:46.750+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:47.169 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:47.168+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:47.651 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:47.650+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:47.754 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:47.753+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:48.153 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:48.153+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:48.641 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:48.640+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:48.742 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:48.741+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:49.155 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:49.154+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:49.690 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:49.689+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:49.727 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:49.726+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:50.119 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:50.119+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:50.685 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:50.686+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:50.757 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:50.758+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:51.112 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:51.113+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:51.678 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:51.677+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:51.730 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:51.730+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:52.077 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:52.077+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:52.692 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:52.692+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:52.760 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:52.760+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:53.051 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:53.051+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:53.661 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:53.661+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:53.778 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:53.778+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:54.028 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:54.029+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:54.670 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:54.670+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:54.737 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:54.738+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:55.048 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:55.049+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:55.686 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:55.686+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:55.776 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:55.777+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:56.082 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:56.083+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:56.692 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:56.692+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:56.819 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:56.820+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:57.088 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:57.088+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:57.668 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:57.669+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:57.833 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:57.833+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:58.135 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:58.136+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:58.713 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:58.713+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:58.840 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:58.840+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:59.097 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T19:59:59.097+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:59.743 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T19:59:59.743+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T19:59:59.795 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T19:59:59.795+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:00.095 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:00.095+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:00.717 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:00.718+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:00.752 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:00.753+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:01.081 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:01.081+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:01.711 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:01.711+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:01.763 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:01.764+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:02.096 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:02.097+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:02.701 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:02.702+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:02.727 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:02.728+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:03.141 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:03.141+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:03.708 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:03.707+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:03.736 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:03.736+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:04.149 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:04.147+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:04.736 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:04.735+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:04.752 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:04.751+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:05.170 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:05.170+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:05.740 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:05.739+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:05.769 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:05.769+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:06.158 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:06.157+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:06.699 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:06.698+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:06.782 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:06.781+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:07.147 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:07.146+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:07.739 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:07.738+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:07.748 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:07.748+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:08.184 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:08.183+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:08.724 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:08.723+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:08.751 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:08.750+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:09.234 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:09.233+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:09.701 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:09.700+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:09.779 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:09.778+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:10.260 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:10.260+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:10.743 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:10.742+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:10.826 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:10.825+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:11.268 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:11.267+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:11.695 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:11.695+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:11.844 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:11.843+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:12.263 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:12.263+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:12.653 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:12.652+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:12.821 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:12.821+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:13.291 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:13.290+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:13.618 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:13.618+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:13.780 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:13.779+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:14.301 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:14.300+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:14.599 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:14.598+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:14.793 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:14.792+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:15.281 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:15.281+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:15.553 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:15.552+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:15.822 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:15.821+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:16.317 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:16.316+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:16.602 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:16.602+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:16.778 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:16.778+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:17.292 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:17.292+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:17.603 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:17.603+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:17.824 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:17.823+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:18.320 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:18.319+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:18.650 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:18.649+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:18.826 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:18.825+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:19.273 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:19.272+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:19.698 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:19.698+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:19.797 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:19.797+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:20.261 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:20.261+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:20.717 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:20.717+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:20.828 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:20.827+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:21.306 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:21.305+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:21.679 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:21.679+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:21.863 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:21.862+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:22.337 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:22.336+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:22.679 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:22.679+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:22.819 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:22.818+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:23.374 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:23.374+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:23.708 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:23.707+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:23.817 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:23.817+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:24.394 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:24.394+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:24.758 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:24.757+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:24.771 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:24.771+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:25.430 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:25.429+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:25.772 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:25.771+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:25.796 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:25.795+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:26.403 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:26.402+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:26.755 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:26.754+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:26.842 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:26.842+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:27.428 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:27.428+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:27.741 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:27.740+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:27.875 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:27.874+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:28.449 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:28.448+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:28.714 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:28.713+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:28.841 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:28.840+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:29.461 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:29.460+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:29.693 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:29.692+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:29.884 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:29.884+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:30.477 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:30.476+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:30.709 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:30.708+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:30.865 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:30.864+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:31.468 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:31.467+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:31.741 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:31.740+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:31.819 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:31.819+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:32.517 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:32.517+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:32.713 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:32.713+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:32.830 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:32.829+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:33.529 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:33.530+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:33.755 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:33.755+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:33.833 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:33.832+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:34.553 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:34.554+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:34.782 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:34.781+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:34.832 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:34.832+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:35.551 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:35.551+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:35.740 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:35.740+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:35.828 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:35.829+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:36.554 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:36.554+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:36.725 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:36.725+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:36.818 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:36.819+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:37.505 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:37.505+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:37.683 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:37.683+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:37.806 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:37.807+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:38.506 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:38.506+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:38.670 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:38.670+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:38.833 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:38.833+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:39.521 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:39.521+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:39.645 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:39.645+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:39.829 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:39.829+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:40.528 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:40.529+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:40.648 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:40.649+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:40.880 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:40.880+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:41.561 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:41.562+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:41.660 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:41.661+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:41.878 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:41.878+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:42.594 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:42.594+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:42.687 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:42.687+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:42.859 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:42.859+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:43.561 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:43.561+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:43.684 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:43.685+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:43.838 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:43.838+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:44.604 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:44.604+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:44.672 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:44.673+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:44.872 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:44.872+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:45.654 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:45.654+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:45.654 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:45.655+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:45.828 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:45.829+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:46.669 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:46.669+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:46.670 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:46.671+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:46.780 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:46.780+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:47.623 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:47.624+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:47.674 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:47.674+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:47.822 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:47.822+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:48.580 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:48.579+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:48.670 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:48.669+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:48.816 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:48.815+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:49.599 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:49.599+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:49.679 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:49.678+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:49.831 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:49.830+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:50.604 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:50.603+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:50.685 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:50.685+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:50.871 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:50.870+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:51.638 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:51.633+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:51.653 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:51.653+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:51.912 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:51.911+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:52.613 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:52.613+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:52.676 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:52.675+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:52.921 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:52.920+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:53.647 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:53.646+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:53.652 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:53.652+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:53.896 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:53.896+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:54.665 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:54.665+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:54.691 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:54.691+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:54.939 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:54.938+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:55.707 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:55.706+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:55.709 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:55.708+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:55.916 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:55.915+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:56.676 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:56.675+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:56.749 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:56.749+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:56.962 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:56.961+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:57.641 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:57.640+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:57.774 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:57.774+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:57.939 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:57.938+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:58.614 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:58.613+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:58.821 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:58.821+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:58.956 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:00:58.954+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:59.606 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:00:59.606+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:00:59.833 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:00:59.832+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:00.002 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:00.002+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:00.592 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:00.592+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:00.818 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:00.818+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:01.052 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:01.051+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:01.615 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:01.614+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:01.787 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:01.786+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:02.054 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:02.054+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:02.608 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:02.608+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:02.821 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:02.821+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:03.033 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:03.033+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:03.574 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:03.573+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:03.859 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:03.858+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:04.082 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:04.082+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:04.623 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:04.622+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:04.833 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:04.832+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:05.067 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:05.066+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:05.654 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:05.653+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:05.832 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:05.831+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:06.049 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:06.048+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:06.673 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:06.672+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:06.826 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:06.825+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:07.032 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:07.032+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:07.717 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:07.716+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:07.870 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:07.870+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:08.035 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:08.035+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:08.765 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:08.764+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:08.882 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:08.881+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:08.990 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:08.989+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:09.738 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:09.738+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:09.920 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:09.919+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:09.979 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:09.978+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:10.765 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:10.764+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:10.932 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:10.931+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:11.018 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:11.017+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:11.730 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:11.730+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:11.908 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:11.908+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:12.008 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:12.007+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:12.700 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:12.700+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:12.909 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:12.908+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:13.006 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:13.005+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:13.748 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:13.747+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:13.907 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:13.906+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:13.966 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:13.965+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:14.775 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:14.775+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:14.861 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:14.861+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:14.958 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:14.957+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:15.765 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:15.765+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:15.812 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:15.811+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:15.985 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:15.985+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:16.795 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:16.795+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:16.852 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:16.852+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:17.001 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:17.001+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:17.811 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:17.811+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:17.876 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:17.876+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:18.032 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:18.031+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:18.783 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:18.783+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:18.902 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:18.902+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:19.000 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:19.000+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:19.783 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:19.784+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:19.873 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:19.874+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:19.981 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:19.982+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:20.765 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:20.766+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:20.824 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:20.825+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:20.962 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:20.962+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:21.807 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:21.808+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:21.816 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:21.816+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:21.967 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:21.967+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:22.772 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:22.772+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:22.839 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:22.839+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:22.942 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:22.942+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:23.729 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:23.729+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:23.797 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:23.798+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:23.908 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:23.909+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:24.751 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:24.751+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:24.834 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:24.835+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:24.905 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:24.905+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:25.790 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:25.790+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:25.871 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:25.871+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:25.875 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:25.876+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:26.776 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:26.776+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:26.897 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:26.897+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:26.920 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:26.920+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:27.727 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:27.727+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:27.862 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:27.863+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:27.949 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:27.949+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:28.678 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:28.679+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:28.875 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:28.876+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:28.912 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:28.913+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:29.667 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:29.668+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:29.871 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:29.871+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:29.892 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:29.892+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:30.677 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:30.677+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:30.837 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:30.837+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:30.872 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:30.872+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:31.652 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:31.653+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:31.811 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:31.811+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:31.875 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:31.875+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:32.661 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:32.661+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:32.769 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:32.768+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:32.848 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:32.847+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:33.685 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:33.684+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:33.785 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:33.784+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:33.893 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:33.892+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:34.636 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:34.635+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:34.752 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:34.752+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:34.856 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:34.855+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:35.599 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:35.599+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:35.735 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:35.735+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:35.826 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:35.825+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:36.625 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:36.624+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:36.767 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:36.767+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:36.783 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:36.783+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:37.650 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:37.649+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:37.745 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:37.745+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:37.764 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:37.764+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:38.648 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:38.647+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:38.703 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:38.702+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:38.735 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:38.735+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:39.615 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:39.614+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:39.677 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:39.677+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:39.689 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:39.689+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:40.585 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:40.584+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:40.693 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:40.693+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:40.723 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:40.722+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:41.612 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:41.611+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:41.709 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:41.708+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:41.736 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:41.735+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:42.626 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:42.625+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:42.706 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:42.705+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:42.739 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:42.739+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:43.612 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:43.612+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:43.718 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:43.717+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:43.743 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:43.743+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:44.584 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:44.584+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:44.724 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:44.723+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:44.790 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:44.789+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:45.558 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:45.557+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:45.714 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:45.713+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:45.783 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:45.782+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:46.540 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:46.537+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:46.751 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:46.750+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:46.791 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:46.790+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:47.584 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:47.583+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:47.774 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:47.773+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:47.779 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:47.779+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:48.617 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:48.617+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:48.742 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:48.742+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:48.821 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:48.820+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:49.651 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:49.650+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:49.753 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:49.752+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:49.813 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:49.811+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:50.612 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:50.612+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:50.779 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:50.779+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:50.783 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:50.783+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:51.600 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:51.599+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:51.828 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:51.827+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:51.834 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:51.834+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:52.556 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:52.555+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:52.819 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:52.819+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:52.851 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:52.850+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:53.597 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:53.596+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:53.783 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:53.782+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:53.862 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:53.862+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:54.610 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:54.609+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:54.811 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:54.811+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:54.899 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:54.899+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:55.579 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:55.578+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:55.806 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:55.805+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:55.860 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:55.859+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:56.627 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:56.626+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:56.780 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:56.779+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:56.885 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:56.885+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:57.665 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:57.664+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:57.778 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:57.777+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:57.920 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:57.919+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:58.694 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:58.693+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:58.749 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:58.748+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:58.955 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:58.954+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:59.699 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:01:59.699+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:59.709 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:01:59.709+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:01:59.985 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:01:59.985+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:00.695 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:00.695+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:00.724 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:00.723+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:00.955 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:00.954+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:01.661 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:01.661+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:01.765 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:01.765+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:01.981 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:01.980+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:02.652 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:02.652+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:02.800 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:02.799+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:02.935 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:02.935+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:03.603 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:03.603+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:03.780 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:03.781+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:03.903 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:03.903+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:04.596 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:04.597+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:04.790 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:04.791+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:04.933 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:04.933+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:05.608 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:05.608+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:05.802 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:05.802+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:05.891 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:05.891+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:06.624 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:06.625+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:06.774 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:06.775+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:06.880 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:06.881+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:07.606 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:07.606+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:07.766 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:07.767+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:07.835 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:07.835+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:08.604 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:08.605+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:08.815 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:08.816+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:08.882 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:08.883+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:09.631 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:09.632+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:09.839 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:09.840+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:09.916 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:09.916+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:10.672 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:10.672+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:10.855 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:10.855+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:10.940 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:10.941+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:11.655 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:11.655+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:11.890 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:11.890+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:11.943 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:11.944+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:12.701 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:12.701+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:12.906 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:12.907+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:12.974 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:12.974+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:13.677 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:13.677+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:13.902 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:13.902+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:13.941 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:13.941+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:14.722 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:14.722+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:14.893 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:14.894+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:14.946 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:14.946+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:15.767 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:15.767+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:15.864 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:15.864+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:15.937 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:15.937+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:16.803 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:16.802+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:16.885 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:16.884+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:16.969 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:16.969+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:17.766 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:17.765+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:17.930 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:17.930+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:17.996 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:17.995+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:18.718 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:18.717+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:18.958 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:18.957+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:18.979 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:18.978+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:19.756 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:19.755+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:19.960 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:19.960+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:19.972 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:19.972+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:20.720 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:20.720+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:20.938 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:20.937+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:20.974 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:20.973+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:21.729 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:21.729+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:21.916 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:21.915+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:21.995 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:21.995+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:22.694 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:22.693+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:22.895 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:22.894+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:23.030 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:23.029+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:23.740 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:23.739+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:23.847 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:23.846+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:24.005 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:24.004+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:24.764 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:24.763+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:24.868 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:24.868+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:25.004 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:25.003+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:25.733 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:25.733+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:25.852 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:25.852+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:25.998 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:25.998+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:26.780 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:26.779+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:26.841 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:26.840+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:26.956 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:26.956+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:27.804 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:27.804+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:27.828 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:27.828+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:27.954 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:27.953+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:28.833 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:28.832+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:28.879 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:28.878+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:28.967 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:28.966+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:29.785 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:29.785+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:29.892 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:29.891+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:29.964 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:29.964+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:30.779 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:30.778+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:30.939 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:30.938+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:30.961 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:30.960+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:31.736 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:31.735+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:31.958 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:31.957+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:31.980 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:31.979+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:32.690 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:32.690+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:32.918 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:32.917+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:32.974 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:32.973+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:33.677 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:33.676+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:33.881 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:33.881+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:33.989 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:33.988+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:34.674 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:34.673+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:34.931 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:34.930+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:34.949 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:34.949+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:35.680 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:35.679+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:35.957 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:35.956+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:35.984 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:35.983+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:36.656 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:36.656+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:36.997 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:36.996+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:37.006 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:37.006+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:37.691 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:37.690+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:38.014 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:38.013+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:38.026 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:38.025+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:38.646 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:38.646+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:38.985 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:38.984+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:39.037 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:39.036+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:39.641 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:39.640+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:40.018 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:40.017+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:40.065 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:40.064+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:40.603 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:40.602+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:40.972 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:40.971+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:41.036 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:41.035+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:41.617 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:41.616+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:41.988 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:41.987+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:41.999 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:41.999+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:42.588 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:42.587+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:42.971 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:42.971+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:42.982 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:42.981+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:43.574 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:43.573+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:43.932 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:43.931+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:43.997 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:43.996+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:44.592 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:44.591+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:44.902 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:44.901+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:45.025 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:45.024+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:45.570 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:45.569+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:45.875 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:45.874+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:46.069 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:46.068+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:46.522 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:46.521+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:46.886 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:46.886+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:47.072 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:47.071+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:47.545 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:47.545+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:47.914 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:47.914+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:48.079 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:48.079+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:48.555 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:48.555+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:48.919 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:48.919+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:49.085 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:49.085+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:49.547 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:49.548+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:49.957 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:49.958+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:50.068 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:50.069+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:50.590 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:50.590+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:50.974 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:50.974+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:51.112 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:51.112+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:51.631 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:51.632+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:52.008 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:52.008+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:52.094 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:52.094+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:52.672 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:52.672+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:52.988 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:52.988+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:53.141 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:53.142+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:53.653 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:53.653+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:53.943 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:53.943+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:54.135 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:54.136+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:54.629 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:54.630+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:54.967 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:54.967+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:55.153 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:55.153+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:55.637 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:55.638+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:55.940 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:55.940+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:56.111 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:56.111+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:56.647 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:56.648+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:56.972 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:56.973+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:57.128 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:57.128+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:57.682 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:57.682+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:57.925 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:57.925+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:58.154 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:58.155+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:58.698 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:58.699+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:58.960 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:02:58.960+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:59.139 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:02:59.139+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:02:59.678 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:02:59.678+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:00.004 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:00.005+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:00.178 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:00.179+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:00.649 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:00.649+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:01.033 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:01.032+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:01.158 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:01.158+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:01.664 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:01.664+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:02.067 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:02.067+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:02.135 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:02.134+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:02.664 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:02.664+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:03.118 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:03.117+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:03.166 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:03.166+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:03.708 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:03.707+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:04.111 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:04.111+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:04.192 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:04.192+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:04.751 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:04.751+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:05.107 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:05.107+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:05.211 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:05.210+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:05.797 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:05.796+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:06.091 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:06.091+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:06.242 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:06.242+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:06.761 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:06.760+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:07.094 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:07.093+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:07.246 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:07.245+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:07.784 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:07.783+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:08.058 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:08.057+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:08.220 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:08.219+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:08.746 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:08.745+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:09.032 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:09.031+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:09.213 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:09.213+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:09.733 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:09.733+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:10.075 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:10.074+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:10.221 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:10.220+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:10.738 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:10.737+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:11.060 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:11.059+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:11.190 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:11.190+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:11.710 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:11.710+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:12.056 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:12.055+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:12.212 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:12.212+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:12.671 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:12.671+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:13.044 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:13.043+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:13.194 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:13.194+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:13.673 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:13.672+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:14.046 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:14.045+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:14.155 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:14.154+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:14.660 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:14.659+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:15.056 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:15.055+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:15.108 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:15.107+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:15.620 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:15.618+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:16.039 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:16.038+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:16.121 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:16.120+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:16.598 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:16.597+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:17.078 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:17.077+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:17.080 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:17.080+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:17.614 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:17.613+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:18.044 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:18.043+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:18.122 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:18.121+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:18.620 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:18.619+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:19.051 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:19.050+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:19.106 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:19.105+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:19.625 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:19.624+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:20.074 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:20.073+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:20.125 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:20.125+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:20.618 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:20.617+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:21.107 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:21.106+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:21.112 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:21.111+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:21.574 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:21.573+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:22.109 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:22.108+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:22.129 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:22.128+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:22.574 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:22.573+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:23.142 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:23.142+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:23.152 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:23.152+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:23.623 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:23.622+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:24.185 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:24.184+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:24.188 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:24.187+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:24.579 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:24.578+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:25.195 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:25.194+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:25.209 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:25.208+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:25.575 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:25.574+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:26.204 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:26.203+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:26.216 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:26.216+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:26.587 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:26.586+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:27.198 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:27.197+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:27.241 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:27.240+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:27.579 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:27.579+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:28.188 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:28.188+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:28.269 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:28.268+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:28.553 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:28.553+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:29.176 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:29.176+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:29.293 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:29.293+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:29.597 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:29.597+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:30.190 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:30.189+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:30.267 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:30.266+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:30.609 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:30.609+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:31.237 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:31.236+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:31.274 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:31.274+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:31.607 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:31.607+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:32.274 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:32.274+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:32.324 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:32.323+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:32.627 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:32.627+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:33.318 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:33.318+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:33.355 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:33.356+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:33.656 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:33.657+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:34.298 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:34.298+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:34.378 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:34.379+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:34.700 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:34.701+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:35.303 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:35.304+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:35.347 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:35.347+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:35.678 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:35.678+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:36.266 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:36.266+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:36.300 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:36.300+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:36.709 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:36.709+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:37.295 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:37.296+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:37.307 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:37.307+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:37.693 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:37.693+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:38.325 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:38.326+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:38.347 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:38.348+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:38.668 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:38.668+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:39.283 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:39.282+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:39.361 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:39.361+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:39.618 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:39.619+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:40.246 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:40.246+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:40.393 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:40.393+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:40.616 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:40.616+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:41.280 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:41.280+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:41.365 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:41.365+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:41.646 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:41.647+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:42.285 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:42.286+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:42.318 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:42.318+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:42.620 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:42.620+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:43.287 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:43.287+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:43.325 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:43.325+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:43.649 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:43.650+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:44.286 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:44.286+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:44.324 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:44.325+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:44.626 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:44.627+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:45.271 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:45.271+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:45.305 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:45.304+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:45.610 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:45.609+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:46.285 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:46.284+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:46.336 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:46.336+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:46.589 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:46.588+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:47.243 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:47.242+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:47.326 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:47.325+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:47.577 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:47.577+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:48.227 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:48.226+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:48.336 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:48.336+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:48.600 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:48.600+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:49.265 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:49.264+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:49.311 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:49.310+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:49.587 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:49.587+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:50.276 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:50.276+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:50.297 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:50.297+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:50.611 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:50.610+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:51.259 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:51.258+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:51.296 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:51.295+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:51.660 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:51.659+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:52.293 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:52.293+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:52.299 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:52.299+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:52.645 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:52.644+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:53.286 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:53.286+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:53.349 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:53.348+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:53.661 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:53.660+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:54.271 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:54.270+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:54.322 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:54.321+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:54.667 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:54.666+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:55.295 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:55.294+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:55.316 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:55.315+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:55.665 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:55.664+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:56.270 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:56.269+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:56.360 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:56.360+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:56.687 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:56.687+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:57.235 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:57.234+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:57.350 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:57.349+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:57.669 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:57.668+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:58.191 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:58.191+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:58.305 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:58.304+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:58.636 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:58.635+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:59.169 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:03:59.169+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:59.279 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:03:59.278+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:03:59.669 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:03:59.668+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:00.147 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:00.146+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:00.262 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:00.261+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:00.702 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:00.702+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:01.188 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:01.188+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:01.244 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:01.244+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:01.706 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:01.702+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:02.198 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:02.198+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:02.249 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:02.248+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:02.684 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:02.683+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:03.211 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:03.210+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:03.260 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:03.259+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:03.636 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:03.636+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:04.189 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:04.189+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:04.306 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:04.306+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:04.673 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:04.672+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:05.141 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:05.141+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:05.348 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:05.348+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:05.680 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:05.679+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:06.173 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:06.172+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:06.347 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:06.346+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:06.692 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:06.691+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:07.180 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:07.179+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:07.395 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:07.394+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:07.726 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:07.725+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:08.173 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:08.173+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:08.390 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:08.390+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:08.679 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:08.678+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:09.164 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:09.163+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:09.397 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:09.396+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:09.644 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:09.643+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:10.161 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:10.161+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:10.364 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:10.363+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:10.608 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:10.608+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:11.158 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:11.157+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:11.354 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:11.353+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:11.650 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:11.649+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:12.127 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:12.127+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:12.375 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:12.375+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:12.608 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:12.607+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:13.163 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:13.162+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:13.360 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:13.357+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:13.616 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:13.615+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:14.207 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:14.207+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:14.349 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:14.348+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:14.593 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:14.590+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:15.215 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:15.214+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:15.364 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:15.364+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:15.549 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:15.548+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:16.167 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:16.166+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:16.410 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:16.409+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:16.579 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:16.579+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:17.174 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:17.174+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:17.392 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:17.393+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:17.567 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:17.568+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:18.213 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:18.214+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:18.429 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:18.429+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:18.587 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:18.587+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:19.177 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:19.178+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:19.469 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:19.469+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:19.598 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:19.598+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:20.140 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:20.139+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:20.423 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:20.423+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:20.636 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:20.636+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:21.120 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:21.120+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:21.426 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:21.426+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:21.666 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:21.667+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:22.108 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:22.108+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:22.453 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:22.454+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:22.714 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:22.714+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:23.074 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:23.074+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:23.474 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:23.474+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:23.679 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:23.680+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:24.122 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:24.122+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:24.524 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:24.524+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:24.702 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:24.702+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:25.132 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:25.132+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:25.483 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:25.483+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:25.710 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:25.710+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:26.165 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:26.166+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:26.486 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:26.486+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:26.672 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:26.672+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:27.184 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:27.184+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:27.456 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:27.457+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:27.715 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:27.715+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:28.156 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:28.157+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:28.501 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:28.501+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:28.715 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:28.716+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:29.136 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:29.136+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:29.528 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:29.529+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:29.690 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:29.689+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:30.133 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:30.133+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:30.579 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:30.579+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:30.670 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:30.669+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:31.086 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:31.085+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:31.557 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:31.556+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:31.691 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:31.690+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:32.129 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:32.128+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:32.581 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:32.579+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:32.677 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:32.676+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:33.138 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:33.138+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:33.541 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:33.541+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:33.641 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:33.640+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:34.110 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:34.109+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:34.494 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:34.493+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:34.622 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:34.622+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:35.142 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:35.142+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:35.526 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:35.526+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:35.657 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:35.656+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:36.149 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:36.149+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:36.492 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:36.492+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:36.625 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:36.624+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:37.163 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:37.162+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:37.494 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:37.493+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:37.616 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:37.616+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:38.133 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:38.133+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:38.535 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:38.535+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:38.621 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:38.621+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:39.145 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:39.145+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:39.490 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:39.489+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:39.655 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:39.654+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:40.192 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:40.191+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:40.503 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:40.502+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:40.676 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:40.676+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:41.196 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:41.195+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:41.498 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:41.497+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:41.628 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:41.627+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:42.213 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:42.212+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:42.549 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:42.548+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:42.596 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:42.595+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:43.213 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:43.213+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:43.565 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:43.565+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:43.583 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:43.583+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:44.250 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:44.249+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:44.597 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:44.597+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:44.607 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:44.607+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 16 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:45.291 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:45.290+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:45.578 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:45.578+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:45.643 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:45.642+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:46.249 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:46.249+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:46.546 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:46.546+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:46.648 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:46.647+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:47.230 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:47.230+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:47.596 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:47.595+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:47.606 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:47.606+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:48.195 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:48.195+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:48.575 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:48.574+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:48.643 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:48.642+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:49.172 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:49.171+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:49.611 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:49.610+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:49.648 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:49.647+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:50.208 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:50.207+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:50.619 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:50.619+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:50.634 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:50.634+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:51.241 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:51.240+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:51.621 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:51.620+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:51.641 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:51.639+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:52.233 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:52.232+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:52.609 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:52.609+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:52.614 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:52.614+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:53.279 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:53.278+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:53.580 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:53.579+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:53.639 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:53.638+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:54.301 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:54.300+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:54.586 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:54.585+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:54.660 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:54.659+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:55.349 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:55.348+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:55.604 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:55.603+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:55.632 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:55.631+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:56.361 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:56.360+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:56.626 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:56.625+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:56.628 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:56.628+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:57.348 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:57.347+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:57.597 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:57.595+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:57.634 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:57.633+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:58.304 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:58.303+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:58.582 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:58.581+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:58.621 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:58.621+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:59.325 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:04:59.324+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:59.561 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:04:59.560+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:04:59.658 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:04:59.658+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:00.350 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:00.350+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:00.586 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:00.585+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:00.662 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:00.663+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:01.394 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:01.395+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:01.564 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:01.564+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:01.684 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:01.684+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:02.386 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:02.386+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:02.545 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:02.545+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:02.721 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:02.722+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:03.398 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:03.399+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:03.534 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:03.534+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:03.684 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:03.685+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:04.388 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:04.388+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:04.584 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:04.584+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:04.684 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:04.684+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:05.406 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:05.407+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:05.630 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:05.630+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:05.711 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:05.711+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:06.394 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:06.394+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:06.602 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:06.602+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:06.733 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:06.733+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:07.351 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:07.351+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:07.555 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:07.555+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:07.778 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:07.779+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:08.393 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:08.394+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:08.550 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:08.551+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:08.792 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:08.792+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:09.416 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:09.417+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:09.562 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:09.562+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:09.749 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:09.749+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:10.383 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:10.384+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:10.526 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:10.526+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:10.729 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:10.729+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:11.412 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:11.412+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:11.542 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:11.542+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:11.680 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:11.680+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:12.411 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:12.411+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:12.517 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:12.518+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:12.676 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:12.676+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:13.457 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:13.457+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:13.565 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:13.565+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:13.646 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:13.647+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:14.424 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:14.423+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:14.521 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:14.521+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:14.694 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:14.693+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:15.413 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:15.412+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:15.503 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:15.502+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:15.743 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:15.742+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:16.374 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:16.374+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:16.461 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:16.461+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:16.731 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:16.731+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:17.422 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:17.421+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:17.467 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:17.466+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:17.776 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:17.775+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:18.465 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:18.465+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:18.473 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:18.472+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:18.787 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:18.786+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:19.464 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:19.463+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:19.478 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:19.478+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:19.804 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:19.804+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:20.482 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:20.481+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:20.515 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:20.514+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:20.850 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:20.849+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:21.489 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:21.488+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:21.554 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:21.554+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:21.857 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:21.857+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:22.468 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:22.467+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:22.603 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:22.602+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:22.900 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:22.899+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:23.446 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:23.446+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:23.562 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:23.561+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:23.936 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:23.935+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:24.492 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:24.492+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:24.559 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:24.559+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:24.967 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:24.967+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:25.447 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:25.446+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:25.597 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:25.596+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:26.007 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:26.006+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:26.492 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:26.491+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:26.632 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:26.631+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:27.015 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:27.015+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:27.464 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:27.463+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:27.672 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:27.671+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:27.982 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:27.981+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:28.418 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:28.417+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:28.697 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:28.697+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:29.005 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:29.004+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:29.370 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:29.369+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:29.709 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:29.708+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:30.019 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:30.018+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:30.376 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:30.375+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:30.717 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:30.716+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:31.033 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:31.032+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:31.424 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:31.423+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:31.719 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:31.718+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:32.008 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:32.008+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:32.379 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:32.379+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:32.729 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:32.728+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:32.989 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:32.988+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:33.365 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:33.364+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:33.726 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:33.726+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:34.030 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:34.030+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:34.360 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:34.359+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:34.746 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:34.745+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:35.071 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:35.071+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:35.360 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:35.359+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:35.702 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:35.701+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:36.076 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:36.076+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:36.398 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:36.397+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:36.669 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:36.668+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:37.047 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:37.047+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:37.395 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:37.394+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:37.689 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:37.689+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:38.055 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:38.054+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:38.433 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:38.432+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:38.663 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:38.662+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:39.005 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:39.004+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:39.391 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:39.390+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:39.693 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:39.692+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:40.046 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:40.046+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:40.435 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:40.434+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:40.677 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:40.676+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:41.042 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:41.041+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:41.403 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:41.403+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:41.669 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:41.669+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:42.061 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:42.061+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:42.374 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:42.374+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:42.694 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:42.694+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:43.036 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:43.035+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:43.411 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:43.411+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:43.718 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:43.718+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:44.007 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:44.007+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:44.439 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:44.439+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:44.735 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:44.735+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:45.019 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:45.019+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:45.433 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:45.433+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:45.762 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:45.762+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:46.033 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:46.034+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:46.417 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:46.418+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:46.743 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:46.743+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:47.071 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:47.071+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:47.421 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:47.421+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:47.700 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:47.700+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:48.047 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:48.047+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:48.419 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:48.419+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:48.750 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:48.750+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:49.088 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:49.088+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:49.391 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:49.391+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:49.782 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:49.782+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:50.070 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:50.070+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:50.382 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:50.382+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:50.817 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:50.817+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:51.032 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:51.030+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:51.372 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:51.372+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:51.802 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:51.802+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:52.061 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:52.061+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:52.387 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:52.388+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:52.835 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:52.836+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:53.060 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:53.061+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:53.369 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:53.369+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:53.867 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:53.868+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:54.084 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:54.085+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:54.339 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:54.339+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:54.823 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:54.823+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:55.119 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:55.119+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:55.296 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:55.296+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:55.797 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:55.798+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:56.078 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:56.079+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:56.309 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:56.309+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:56.838 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:56.839+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:57.067 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:57.067+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:57.321 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:57.318+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:57.885 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:57.886+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:58.075 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:58.076+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:58.352 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:58.351+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:58.841 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:58.840+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:59.119 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:05:59.119+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:59.385 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:05:59.385+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:05:59.837 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:05:59.836+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:00.156 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:00.155+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:00.358 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:00.357+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:00.802 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:00.801+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:01.146 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:01.146+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:01.315 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:01.315+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:01.825 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:01.824+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:02.101 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:02.101+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:02.271 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:02.271+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:02.861 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:02.860+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:03.146 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:03.145+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:03.287 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:03.287+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:03.863 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:03.861+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:04.189 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:04.188+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:04.288 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:04.286+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:04.848 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:04.845+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:05.211 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:05.211+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:05.249 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:05.248+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:05.853 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:05.853+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:06.195 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:06.195+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:06.229 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:06.228+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:06.839 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:06.838+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:07.222 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:07.221+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:07.243 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:07.242+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:07.793 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:07.792+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:08.199 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:08.198+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:08.211 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:08.211+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:08.818 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:08.815+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:09.241 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:09.240+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:09.245 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:09.245+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:09.851 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:09.851+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:10.229 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:10.228+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:10.277 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:10.276+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:10.833 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:10.833+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:11.276 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:11.275+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:11.301 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:11.300+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:11.841 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:11.839+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:12.269 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:12.268+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:12.286 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:12.286+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:12.888 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:12.888+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:13.247 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:13.247+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:13.268 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:13.267+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:13.881 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:13.880+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:14.233 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:14.232+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:14.239 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:14.238+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:14.856 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:14.856+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:15.253 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:15.252+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:15.265 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:15.264+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:15.903 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:15.903+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:16.239 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:16.238+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:16.260 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:16.260+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:16.920 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:16.919+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:17.269 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:17.268+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:17.284 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:17.284+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:17.910 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:17.910+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:18.311 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:18.310+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:18.324 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:18.323+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:18.864 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:18.863+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:19.306 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:19.305+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:19.348 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:19.348+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:19.859 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:19.857+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:20.296 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:20.295+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:20.336 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:20.335+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:20.845 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:20.845+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:21.312 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:21.311+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:21.373 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:21.370+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:21.800 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:21.799+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:22.313 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:22.312+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:22.336 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:22.335+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:22.838 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:22.837+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:23.287 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:23.286+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:23.350 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:23.350+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:23.803 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:23.803+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:24.329 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:24.326+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:24.396 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:24.395+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:24.814 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:24.814+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:25.333 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:25.332+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:25.443 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:25.442+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:25.794 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:25.793+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:26.310 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:26.310+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:26.395 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:26.394+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:26.802 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:26.802+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:27.289 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:27.289+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:27.415 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:27.415+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:27.848 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:27.847+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:28.337 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:28.338+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:28.441 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:28.440+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:28.800 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:28.800+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:29.381 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:29.380+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:29.438 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:29.438+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:29.801 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:29.802+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:30.344 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:30.345+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:30.424 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:30.425+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:30.848 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:30.849+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:31.340 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:31.341+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:31.449 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:31.450+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:31.823 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:31.824+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:32.312 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:32.313+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:32.458 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:32.459+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:32.848 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:32.848+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:33.320 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:33.321+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:33.464 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:33.464+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:33.887 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:33.887+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:34.346 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:34.346+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:34.436 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:34.436+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:34.856 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:34.856+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:35.347 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:35.347+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:35.462 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:35.463+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:35.809 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:35.809+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:36.371 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:36.371+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:36.433 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:36.431+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:36.788 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:36.788+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:37.355 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:37.355+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:37.443 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:37.443+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:37.823 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:37.823+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:38.384 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:38.384+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:38.426 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:38.426+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:38.780 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:38.781+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:39.382 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:39.382+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:39.438 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:39.438+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:39.745 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:39.746+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:40.396 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:40.397+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:40.477 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:40.478+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:40.704 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:40.704+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:41.422 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:41.423+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:41.514 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:41.514+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:41.734 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:41.734+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:42.462 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:42.462+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:42.560 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:42.560+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:42.700 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:42.699+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:43.424 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:43.423+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:43.585 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:43.584+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:43.671 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:43.671+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:44.395 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:44.394+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:44.601 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:44.600+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:44.697 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:44.697+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:45.427 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:45.426+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:45.573 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:45.572+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:45.708 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:45.707+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:46.471 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:46.470+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:46.604 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:46.603+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:46.674 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:46.673+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:47.481 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:47.481+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:47.590 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:47.589+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:47.706 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:47.705+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:48.497 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:48.496+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:48.633 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:48.632+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:48.689 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:48.688+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:49.475 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:49.474+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:49.612 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:49.611+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:49.663 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:49.662+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:50.518 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:50.517+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:50.564 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:50.563+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:50.694 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:50.694+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:51.537 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:51.531+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:51.603 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:51.602+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:51.696 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:51.695+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:52.541 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:52.540+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:52.593 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:52.591+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:52.669 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:52.669+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:53.549 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:53.549+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:53.631 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:53.631+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:53.668 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:53.668+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:54.555 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:54.554+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:54.641 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:54.640+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:54.669 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:54.668+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:55.540 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:55.540+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:55.625 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:55.624+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:55.695 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:55.694+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:56.536 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:56.535+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:56.583 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:56.582+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:56.719 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:56.714+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:57.530 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:57.530+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:57.584 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:57.583+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:57.726 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:57.726+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:58.492 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:58.491+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:58.629 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:58.628+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:58.708 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:58.708+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:59.510 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:06:59.510+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:59.596 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:06:59.595+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:06:59.660 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:06:59.660+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:00.480 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:00.479+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:00.589 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:00.586+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:00.679 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:00.678+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:01.521 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:01.521+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:01.606 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:01.605+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:01.664 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:01.663+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:02.531 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:02.530+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:02.637 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:02.636+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:02.685 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:02.684+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:03.522 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:03.521+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:03.653 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:03.652+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:03.673 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:03.672+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:04.494 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:04.493+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:04.681 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:04.680+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:04.687 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:04.686+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:05.511 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:05.510+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:05.639 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:05.639+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:05.702 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:05.701+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:06.554 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:06.553+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:06.687 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:06.687+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:06.722 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:06.721+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:07.514 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:07.513+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:07.664 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:07.663+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:07.727 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:07.727+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:08.558 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:08.553+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:08.694 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:08.693+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:08.708 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:08.708+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:09.531 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:09.531+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:09.661 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:09.660+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:09.664 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:09.662+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:10.525 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:10.525+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:10.629 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:10.629+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:10.663 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:10.662+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:11.557 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:11.557+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:11.610 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:11.610+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:11.620 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:11.619+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:12.516 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:12.515+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:12.603 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:12.602+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:12.622 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:12.621+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:13.506 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:13.506+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:13.558 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:13.558+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:13.611 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:13.611+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:14.511 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:14.511+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:14.552 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:14.553+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:14.576 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:14.576+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:15.465 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:15.465+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:15.519 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:15.520+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:15.565 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:15.565+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:16.481 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:16.481+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:16.510 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:16.510+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:16.557 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:16.557+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:17.446 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:17.446+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:17.519 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:17.519+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:17.536 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:17.535+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:18.450 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:18.451+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:18.545 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:18.545+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:18.549 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:18.549+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:19.469 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:19.469+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:19.565 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:19.566+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:19.571 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:19.571+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:20.437 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:20.434+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:20.524 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:20.525+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:20.526 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:20.526+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:21.463 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:21.462+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:21.530 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:21.530+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:21.564 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:21.564+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:22.464 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:22.463+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:22.556 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:22.557+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:22.562 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:22.558+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:23.494 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:23.495+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:23.555 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:23.555+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:23.560 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:23.560+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:24.525 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:24.525+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:24.539 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:24.539+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:24.593 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:24.592+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:25.516 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:25.517+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:25.535 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:25.535+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:25.627 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:25.628+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:26.538 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:26.539+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:26.569 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:26.569+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:26.669 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:26.669+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:27.527 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:27.526+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:27.602 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:27.581+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:27.631 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:27.629+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:28.550 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:28.550+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:28.565 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:28.564+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:28.628 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:28.627+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:29.531 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:29.530+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:29.577 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:29.576+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:29.593 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:29.592+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:30.492 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:30.491+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:30.575 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:30.575+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:30.606 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:30.606+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:31.508 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:31.507+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:31.602 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:31.601+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:31.629 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:31.628+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:32.540 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:32.539+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:32.612 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:32.611+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:32.617 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:32.614+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:33.562 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:33.562+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:33.565 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:33.564+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:33.615 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:33.614+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:34.536 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:34.536+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:34.588 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:34.588+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:34.608 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:34.607+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:35.504 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:35.504+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:35.544 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:35.544+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:35.650 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:35.650+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:36.459 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:36.458+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:36.537 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:36.536+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:36.635 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:36.635+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:37.500 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:37.499+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:37.517 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:37.516+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:37.617 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:37.616+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:38.484 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:38.483+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:38.537 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:38.536+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:38.617 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:38.614+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:39.532 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:39.531+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:39.532 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:39.531+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:39.624 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:39.623+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:40.522 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:40.521+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:40.578 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:40.578+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:40.602 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:40.601+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:41.565 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:41.564+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:41.606 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:41.605+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:41.636 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:41.636+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:42.600 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:42.599+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:42.615 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:42.614+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:42.616 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:42.616+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:43.574 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:43.573+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:43.581 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:43.580+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:43.647 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:43.647+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:44.530 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:44.530+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:44.535 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:44.535+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:44.677 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:44.676+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:45.557 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:45.556+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:45.565 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:45.565+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:45.693 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:45.692+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:46.565 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:46.564+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:46.584 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:46.584+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:46.658 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:46.657+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:47.560 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:47.559+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:47.598 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:47.598+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:47.652 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:47.651+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:48.552 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:48.551+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:48.648 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:48.648+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:48.662 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:48.662+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:49.541 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:49.540+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:49.638 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:49.637+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:49.696 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:49.695+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:50.559 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:50.558+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:50.599 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:50.598+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:50.740 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:50.740+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:51.591 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:51.590+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:51.628 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:51.628+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:51.748 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:51.747+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:52.571 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:52.570+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:52.611 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:52.610+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:52.766 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:52.766+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:53.557 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:53.556+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:53.632 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:53.632+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:53.748 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:53.747+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:54.584 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:54.584+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:54.585 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:54.584+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:54.797 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:54.796+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:55.538 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:55.538+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:55.611 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:55.610+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:55.776 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:55.776+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:56.517 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:56.516+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:56.598 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:56.597+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:56.734 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:56.733+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:57.519 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:57.519+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:57.553 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:57.553+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:57.733 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:57.733+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:58.518 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:58.519+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:58.528 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:58.529+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:58.730 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:58.730+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:59.507 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:07:59.507+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:59.559 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:07:59.558+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:07:59.692 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:07:59.692+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:00.486 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:00.486+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:00.547 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:00.548+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:00.652 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:00.653+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:01.515 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:01.515+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:01.588 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:01.588+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:01.617 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:01.618+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:02.558 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:02.558+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:02.596 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:02.596+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:02.628 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:02.629+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:03.568 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:03.569+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:03.570 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:03.571+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:03.598 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:03.599+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:04.539 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:04.539+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:04.609 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:04.610+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:04.612 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:04.613+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:05.582 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:05.582+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:05.589 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:05.589+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:05.591 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:05.591+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:06.594 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:06.594+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:06.612 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:06.612+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:06.641 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:06.642+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:07.545 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:07.545+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:07.578 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:07.578+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:07.656 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:07.657+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:08.501 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:08.501+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:08.582 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:08.583+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:08.635 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:08.635+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:09.464 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:09.464+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:09.586 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:09.586+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:09.622 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:09.623+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:10.419 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:10.419+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:10.579 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:10.579+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:10.615 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:10.615+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:11.408 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:11.407+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:11.539 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:11.539+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:11.644 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:11.644+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:12.421 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:12.420+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:12.559 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:12.558+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:12.683 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:12.682+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:13.429 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:13.427+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:13.544 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:13.543+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:13.641 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:13.640+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:14.395 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:14.394+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:14.586 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:14.586+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:14.658 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:14.656+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:15.419 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:15.418+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:15.564 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:15.563+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:15.652 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:15.652+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:16.399 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:16.398+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:16.555 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:16.554+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:16.638 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:16.638+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:17.369 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:17.368+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:17.555 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:17.554+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:17.634 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:17.633+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:18.398 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:18.397+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:18.520 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:18.520+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:18.603 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:18.602+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:19.360 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:19.359+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:19.490 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:19.489+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:19.560 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:19.560+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:20.353 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:20.352+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:20.444 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:20.443+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:20.550 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:20.550+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:21.379 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:21.379+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:21.401 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:21.401+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:21.582 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:21.582+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:22.335 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:22.335+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:22.393 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:22.392+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:22.617 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:22.616+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:23.349 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:23.348+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:23.424 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:23.424+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:23.618 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:23.618+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:24.361 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:24.360+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:24.435 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:24.434+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:24.659 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:24.658+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:25.331 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:25.331+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:25.429 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:25.428+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:25.702 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:25.702+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:26.332 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:26.331+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:26.445 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:26.445+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:26.727 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:26.726+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:27.322 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:27.321+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:27.421 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:27.420+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:27.750 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:27.750+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:28.368 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:28.367+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:28.444 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:28.444+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:28.750 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:28.750+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:29.367 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:29.367+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:29.482 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:29.481+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:29.785 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:29.781+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:30.348 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:30.346+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:30.440 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:30.438+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:30.809 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:30.808+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:31.303 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:31.302+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:31.427 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:31.425+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:31.817 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:31.816+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:32.332 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:32.331+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:32.417 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:32.416+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:32.769 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:32.768+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:33.374 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:33.373+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:33.380 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:33.380+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:33.746 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:33.746+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:34.329 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:34.328+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:34.421 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:34.421+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:34.782 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:34.781+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:35.326 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:35.324+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:35.437 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:35.436+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:35.803 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:35.803+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:36.310 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:36.310+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:36.453 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:36.452+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:36.802 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:36.802+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:37.288 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:37.288+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:37.469 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:37.468+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:37.843 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:37.842+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:38.280 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:38.280+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:38.427 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:38.426+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:38.836 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:38.835+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:39.235 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:39.235+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:39.468 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:39.468+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:39.836 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:39.836+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:40.207 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:40.207+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:40.432 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:40.432+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:40.831 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:40.830+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:41.229 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:41.229+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:41.403 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:41.403+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:41.791 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:41.791+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:42.192 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:42.191+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:42.412 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:42.411+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:42.784 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:42.785+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:43.228 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:43.228+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:43.392 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:43.392+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:43.796 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:43.797+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:44.191 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:44.191+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:44.409 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:44.409+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:44.841 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:44.842+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:45.234 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:45.234+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:45.426 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:45.426+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:45.803 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:45.804+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:46.242 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:46.242+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:46.446 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:46.447+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:46.811 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:46.811+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:47.226 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:47.226+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:47.411 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:47.411+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:47.844 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:47.845+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:48.180 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:48.181+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:48.418 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:48.418+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:48.804 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:48.805+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:49.195 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:49.195+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:49.370 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:49.370+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:49.829 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:49.830+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:50.162 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:50.163+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:50.388 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:50.388+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:50.880 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:50.881+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:51.157 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:51.157+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:51.425 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:51.426+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:51.903 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:51.903+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:52.115 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:52.115+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:52.395 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:52.395+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:52.909 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:52.910+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:53.164 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:53.165+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:53.359 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:53.359+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:53.863 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:53.864+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:54.127 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:54.127+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:54.330 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:54.330+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:54.861 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:54.861+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:55.100 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:55.101+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:55.311 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:55.312+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:55.910 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:55.910+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:56.073 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:56.072+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:56.280 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:56.280+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:56.862 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:56.861+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:57.100 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:57.100+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:57.304 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:57.303+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:57.843 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:57.842+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:58.123 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:58.123+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:58.340 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:58.339+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:58.860 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:58.860+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:59.086 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:08:59.085+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:59.369 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:08:59.368+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:08:59.859 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:08:59.858+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:00.123 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:00.122+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:00.396 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:00.395+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:00.841 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:00.840+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:01.088 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:01.088+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:01.405 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:01.404+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:01.818 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:01.817+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:02.073 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:02.073+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:02.372 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:02.372+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:02.807 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:02.806+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:03.036 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:03.036+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:03.373 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:03.372+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:03.795 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:03.794+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:04.083 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:04.083+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:04.346 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:04.345+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:04.778 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:04.778+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:05.110 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:05.110+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:05.351 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:05.351+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:05.733 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:05.732+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:06.099 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:06.098+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:06.332 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:06.331+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:06.731 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:06.730+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:07.078 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:07.077+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:07.346 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:07.345+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:07.743 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:07.742+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:08.104 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:08.103+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:08.357 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:08.355+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:08.709 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:08.709+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:09.143 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:09.142+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:09.326 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:09.326+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:09.757 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:09.756+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:10.153 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:10.152+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:10.352 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:10.351+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:10.738 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:10.737+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:11.125 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:11.124+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:11.387 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:11.386+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:11.704 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:11.703+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:12.114 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:12.114+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:12.394 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:12.393+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:12.684 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:12.683+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:13.119 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:13.119+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:13.417 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:13.416+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:13.730 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:13.729+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:14.084 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:14.083+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:14.383 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:14.382+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:14.753 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:14.753+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:15.115 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:15.114+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:15.366 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:15.366+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:15.722 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:15.722+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:16.087 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:16.086+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:16.375 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:16.375+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:16.749 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:16.748+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:17.043 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:17.042+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:17.397 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:17.396+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:17.715 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:17.714+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:18.006 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:18.006+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:18.390 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:18.389+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:18.712 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:18.711+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:18.987 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:18.986+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:19.378 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:19.377+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:19.701 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:19.700+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:19.943 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:19.942+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:20.341 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:20.340+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:20.651 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:20.651+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:20.934 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:20.934+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:21.359 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:21.359+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:21.637 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:21.637+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:21.979 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:21.978+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:22.397 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:22.396+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:22.620 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:22.620+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:23.024 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:23.024+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:23.386 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:23.385+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:23.593 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:23.590+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:24.072 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:24.071+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:24.369 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:24.368+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:24.631 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:24.630+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:25.108 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:25.107+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:25.367 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:25.366+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:25.597 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:25.596+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:26.094 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:26.093+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:26.346 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:26.346+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:26.571 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:26.571+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:27.062 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:27.062+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:27.328 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:27.328+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:27.593 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:27.593+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:28.068 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:28.067+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:28.321 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:28.321+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:28.609 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:28.610+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:29.046 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:29.046+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:29.275 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:29.275+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:29.626 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:29.626+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:30.039 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:30.040+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:30.233 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:30.233+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:30.609 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:30.609+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:30.997 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:30.997+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:31.280 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:31.280+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:31.596 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:31.597+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:31.995 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:31.995+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:32.248 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:32.249+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:32.647 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:32.648+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:32.991 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:32.992+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:33.253 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:33.253+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:33.659 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:33.660+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:33.983 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:33.984+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:34.211 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:34.211+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:34.614 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:34.614+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:34.973 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:34.973+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:35.206 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:35.206+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:35.662 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:35.662+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:35.949 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:35.950+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:36.166 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:36.166+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:36.643 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:36.644+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:36.946 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:36.945+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:37.181 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:37.181+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:37.620 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:37.620+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:37.900 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:37.899+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:38.148 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:38.147+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:38.645 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:38.645+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:38.930 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:38.930+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:39.097 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:39.098+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:39.623 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:39.624+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:39.905 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:39.906+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:40.106 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:40.106+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:40.614 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:40.613+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:40.955 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:40.954+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:41.121 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:41.120+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:41.575 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:41.575+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:41.969 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:41.969+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:42.087 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:42.087+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:42.589 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:42.589+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:43.017 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:43.016+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:43.068 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:43.068+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:43.600 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:43.599+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:44.039 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:44.039+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:44.077 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:44.076+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:44.610 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:44.609+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:45.044 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:45.043+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:45.125 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:45.125+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:45.565 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:45.565+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:46.013 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:46.012+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:46.165 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:46.164+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:46.606 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:46.605+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:46.970 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:46.969+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:47.132 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:47.132+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:47.596 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:47.595+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:47.981 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:47.981+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:48.138 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:48.137+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:48.588 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:48.587+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:48.973 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:48.972+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:49.144 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:49.143+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:49.586 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:49.585+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:49.971 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:49.970+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:50.130 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:50.129+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:50.563 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:50.563+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:50.973 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:50.972+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:51.093 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:51.093+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:51.593 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:51.593+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:51.988 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:51.987+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:52.050 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:52.049+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:52.577 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:52.576+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:53.006 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:53.005+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:53.006 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:53.005+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:53.612 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:53.611+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:53.995 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:53.995+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:54.040 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:54.039+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:54.584 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:54.583+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:54.976 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:54.975+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:55.084 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:55.084+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:55.609 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:55.609+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:55.930 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:55.930+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:56.111 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:56.110+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:56.571 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:56.571+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:56.950 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:56.949+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:57.130 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:57.129+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:57.533 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:57.533+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:57.922 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:57.922+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:58.110 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:58.109+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:58.539 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:58.539+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:58.947 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:58.947+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:59.136 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:09:59.135+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:59.588 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:09:59.588+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:09:59.992 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:09:59.992+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:00.172 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:00.171+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:00.565 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:00.564+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:00.953 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:00.953+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:01.216 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:01.215+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:01.595 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:01.594+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:01.935 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:01.934+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:02.199 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:02.198+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:02.595 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:02.594+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:02.931 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:02.930+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:03.158 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:03.157+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:03.639 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:03.638+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:03.976 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:03.975+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:04.166 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:04.166+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:04.633 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:04.632+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:04.927 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:04.927+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:05.202 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:05.202+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:05.611 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:05.610+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:05.881 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:05.880+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:06.185 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:06.184+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:06.603 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:06.601+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:06.873 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:06.871+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:07.152 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:07.151+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:07.558 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:07.557+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:07.905 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:07.904+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:08.192 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:08.191+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:08.594 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:08.593+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:08.904 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:08.903+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:09.196 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:09.196+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:09.564 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:09.563+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:09.928 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:09.928+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:10.217 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:10.217+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:10.540 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:10.539+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:10.908 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:10.907+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:11.169 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:11.170+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:11.534 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:11.534+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:11.892 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:11.892+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:12.174 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:12.174+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:12.537 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:12.537+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:12.906 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:12.907+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:13.192 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:13.192+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:13.554 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:13.554+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:13.953 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:13.953+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:14.212 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:14.212+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:14.575 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:14.575+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:14.931 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:14.931+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:15.229 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:15.230+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:15.552 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:15.553+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:15.920 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:15.921+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:16.222 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:16.222+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:16.602 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:16.603+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:16.880 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:16.880+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:17.238 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:17.238+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:17.582 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:17.582+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:17.904 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:17.904+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:18.225 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:18.225+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:18.621 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:18.621+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:18.892 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:18.892+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:19.243 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:19.243+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:19.617 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:19.617+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:19.941 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:19.941+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:20.238 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:20.239+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:20.592 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:20.592+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:20.980 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:20.981+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:21.255 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:21.255+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:21.601 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:21.602+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:21.954 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:21.955+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:22.264 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:22.265+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:22.602 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:22.602+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:22.924 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:22.924+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:23.291 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:23.291+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:23.586 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:23.586+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:23.931 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:23.931+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:24.272 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:24.272+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:24.579 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:24.578+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:24.914 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:24.913+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:25.252 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:25.251+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:25.628 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:25.627+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:25.918 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:25.917+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:26.260 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:26.259+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:26.660 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:26.660+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:26.938 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:26.938+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:27.270 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:27.270+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:27.647 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:27.647+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:27.929 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:27.928+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:28.292 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:28.291+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:28.637 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:28.636+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:28.893 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:28.892+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:29.250 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:29.249+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:29.649 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:29.648+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:29.852 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:29.851+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:30.252 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:30.252+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:30.623 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:30.622+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:30.806 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:30.805+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:31.276 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:31.276+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:31.650 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:31.649+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:31.818 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:31.818+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:32.248 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:32.248+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:32.647 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:32.647+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:32.846 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:32.845+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:33.284 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:33.284+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:33.677 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:33.677+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:33.801 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:33.800+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:34.312 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:34.312+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:34.664 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:34.664+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:34.802 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:34.801+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:35.329 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:35.328+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:35.675 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:35.674+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:35.846 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:35.845+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:36.367 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:36.366+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:36.719 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:36.718+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:36.809 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:36.806+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:37.336 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:37.335+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:37.763 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:37.763+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:37.781 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:37.781+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:38.301 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:38.300+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:38.758 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:38.758+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:38.779 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:38.778+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:39.267 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:39.267+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:39.754 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:39.754+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:39.789 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:39.788+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:40.282 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:40.281+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:40.795 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:40.795+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:40.814 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:40.813+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:41.317 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:41.317+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:41.782 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:41.782+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:41.862 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:41.861+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:42.269 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:42.268+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:42.773 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:42.773+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:42.912 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:42.911+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:43.278 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:43.278+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:43.820 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:43.818+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:43.912 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:43.911+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:44.283 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:44.282+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:44.782 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:44.782+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:44.869 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:44.868+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:45.285 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:45.284+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:45.809 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:45.808+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:45.910 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:45.909+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:46.315 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:46.314+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:46.822 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:46.821+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:46.944 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:46.944+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:47.297 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:47.296+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:47.819 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:47.818+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:47.954 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:47.954+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:48.262 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:48.262+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:48.829 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:48.828+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:48.949 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:48.948+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:49.242 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:49.242+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:49.841 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:49.840+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:49.992 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:49.991+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:50.233 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:50.232+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:50.872 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:50.871+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:50.977 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:50.977+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:51.262 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:51.261+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:51.835 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:51.834+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:51.997 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:51.996+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:52.245 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:52.244+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:52.831 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:52.830+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:53.030 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:53.028+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:53.213 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:53.213+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:53.799 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:53.798+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:53.997 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:53.997+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:54.169 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:54.169+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:54.848 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:54.847+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:54.957 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:54.956+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:55.207 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:55.207+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:55.806 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:55.806+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:55.914 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:55.914+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:56.182 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:56.182+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:56.789 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:56.790+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:56.890 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:56.891+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:57.232 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:57.232+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:57.756 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:57.756+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:57.912 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:57.912+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:58.257 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:58.257+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:58.718 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:58.718+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:58.893 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:58.894+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:59.244 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:10:59.244+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:59.718 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:10:59.718+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:10:59.867 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:10:59.868+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:00.285 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:00.285+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:00.687 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:00.688+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:00.896 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:00.896+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:01.280 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:01.281+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:01.682 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:01.682+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:01.866 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:01.867+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:02.243 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:02.244+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:02.715 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:02.716+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:02.895 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:02.896+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:03.267 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:03.267+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:03.681 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:03.681+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:03.920 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:03.920+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:04.243 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:04.243+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:04.663 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:04.663+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:04.927 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:04.928+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:05.230 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:05.231+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:05.702 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:05.702+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:05.939 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:05.939+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:06.196 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:06.196+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:06.698 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:06.698+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:06.973 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:06.973+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:07.220 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:07.221+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:07.724 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:07.725+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:07.947 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:07.948+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:08.204 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:08.204+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:08.728 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:08.727+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:08.952 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:08.951+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:09.194 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:09.193+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:09.773 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:09.772+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:10.001 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:10.001+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:10.209 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:10.208+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:10.776 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:10.776+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:10.968 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:10.967+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:11.215 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:11.214+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:11.763 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:11.763+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:11.966 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:11.965+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:12.253 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:12.252+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:12.726 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:12.721+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:12.939 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:12.938+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:13.278 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:13.277+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:13.716 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:13.715+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:13.945 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:13.944+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:14.240 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:14.239+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:14.725 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:14.724+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:14.956 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:14.955+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:15.205 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:15.204+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:15.750 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:15.749+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:15.983 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:15.983+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:16.240 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:16.239+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:16.746 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:16.745+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:17.006 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:17.005+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:17.222 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:17.222+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:17.768 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:17.767+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:18.055 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:18.054+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:18.210 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:18.209+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:18.741 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:18.740+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:19.070 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:19.069+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:19.200 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:19.199+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:19.733 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:19.732+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:20.101 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:20.100+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:20.218 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:20.218+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:20.742 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:20.742+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:21.110 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:21.109+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:21.206 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:21.205+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:21.733 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:21.732+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:22.142 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:22.141+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:22.195 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:22.194+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:22.712 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:22.712+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:23.102 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:23.101+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:23.204 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:23.204+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:23.740 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:23.740+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:24.095 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:24.094+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:24.186 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:24.185+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:24.769 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:24.768+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:25.081 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:25.080+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:25.203 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:25.203+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:25.818 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:25.817+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:26.055 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:26.054+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:26.159 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:26.158+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:26.798 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:26.797+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:27.015 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:27.015+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:27.143 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:27.143+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:27.827 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:27.826+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:28.054 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:28.053+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:28.107 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:28.106+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:28.841 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:28.840+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:29.038 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:29.037+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:29.138 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:29.137+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:29.823 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:29.822+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:30.007 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:30.006+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:30.157 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:30.157+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:30.813 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:30.812+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:31.007 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:31.007+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:31.137 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:31.136+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:31.798 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:31.797+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:32.040 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:32.040+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:32.164 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:32.163+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:32.787 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:32.786+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:32.996 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:32.995+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:33.133 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:33.132+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:33.821 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:33.820+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:33.957 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:33.956+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:34.112 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:34.111+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:34.835 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:34.835+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:34.934 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:34.933+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:35.143 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:35.142+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:35.871 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:35.871+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:35.948 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:35.947+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:36.148 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:36.147+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:36.905 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:36.865+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:36.978 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:36.977+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:37.198 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:37.198+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:37.822 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:37.822+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:37.966 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:37.966+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:38.233 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:38.232+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:38.803 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:38.803+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:38.925 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:38.925+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:39.209 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:39.208+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:39.821 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:39.820+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:39.941 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:39.939+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:40.259 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:40.260+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:40.772 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:40.772+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:40.899 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:40.899+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:41.289 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:41.289+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:41.789 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:41.790+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:41.854 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:41.854+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:42.280 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:42.280+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:42.818 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:42.818+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:42.884 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:42.884+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:43.324 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:43.325+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:43.781 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:43.781+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:43.898 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:43.898+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:44.337 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:44.337+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:44.785 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:44.785+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:44.904 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:44.905+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:45.313 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:45.313+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:45.805 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:45.805+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:45.950 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:45.950+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:46.325 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:46.326+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:46.793 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:46.794+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:46.946 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:46.946+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:47.358 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:47.358+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:47.755 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:47.755+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:47.910 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:47.910+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:48.402 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:48.403+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:48.806 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:48.806+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:48.940 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:48.940+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:49.420 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:49.420+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:49.822 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:49.823+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:49.976 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:49.976+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:50.405 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:50.406+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:50.873 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:50.873+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:50.949 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:50.950+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:51.360 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:51.361+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:51.852 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:51.852+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:51.962 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:51.962+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:52.379 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:52.379+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:52.832 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:52.832+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:52.987 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:52.988+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:53.366 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:53.366+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:53.824 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:53.823+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:54.025 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:54.025+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:54.322 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:54.321+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:54.863 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:54.855+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:55.063 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:55.062+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:55.361 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:55.360+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:55.821 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:55.820+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:56.045 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:56.044+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:56.407 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:56.407+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:56.866 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:56.865+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:57.026 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:57.025+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:57.374 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:57.373+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:57.858 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:57.857+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:58.046 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:58.046+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:58.374 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:58.373+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:58.821 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:58.820+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:59.014 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:11:59.013+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:59.400 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:11:59.399+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:11:59.781 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:11:59.781+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:00.020 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:00.020+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:00.446 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:00.445+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:00.812 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:00.811+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:01.064 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:01.063+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:01.443 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:01.442+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:01.850 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:01.849+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:02.029 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:02.029+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:02.455 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:02.454+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:02.812 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:02.811+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:03.075 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:03.074+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:03.421 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:03.420+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:03.846 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:03.845+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:04.072 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:04.072+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:04.456 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:04.456+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:04.835 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:04.835+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:05.105 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:05.105+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:05.407 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:05.407+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:05.840 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:05.839+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:06.147 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:06.147+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:06.413 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:06.411+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:06.883 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:06.882+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:07.122 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:07.122+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:07.390 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:07.389+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:07.859 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:07.859+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:08.127 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:08.127+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:08.382 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:08.382+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:08.846 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:08.845+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:09.172 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:09.171+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:09.377 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:09.377+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:09.800 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:09.799+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:10.198 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:10.197+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:10.374 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:10.374+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:10.792 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:10.791+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:11.181 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:11.180+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:11.387 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:11.386+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:11.770 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:11.770+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:12.227 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:12.226+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:12.437 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:12.436+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:12.727 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:12.726+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:13.267 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:13.266+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:13.405 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:13.405+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:13.778 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:13.777+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:14.286 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:14.286+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:14.378 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:14.377+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:14.751 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:14.751+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:15.295 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:15.294+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:15.405 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:15.404+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:15.785 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:15.784+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:16.281 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:16.280+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:16.450 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:16.450+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:16.804 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:16.803+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:17.258 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:17.257+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:17.451 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:17.450+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:17.771 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:17.771+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:18.221 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:18.220+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:18.450 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:18.450+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:18.773 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:18.772+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:19.252 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:19.252+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:19.468 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:19.467+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:19.747 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:19.746+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:20.255 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:20.254+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:20.468 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:20.467+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:20.699 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:20.698+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:21.239 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:21.238+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:21.465 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:21.465+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:21.745 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:21.743+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:22.273 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:22.273+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:22.515 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:22.514+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:22.734 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:22.734+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:23.248 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:23.247+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:23.538 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:23.538+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:23.698 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:23.698+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:24.203 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:24.203+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:24.515 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:24.515+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:24.677 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:24.677+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:25.208 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:25.208+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:25.524 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:25.524+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:25.692 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:25.692+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:26.178 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:26.179+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:26.559 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:26.559+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:26.648 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:26.649+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:27.190 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:27.190+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:27.534 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:27.534+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:27.636 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:27.637+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:28.191 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:28.191+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:28.517 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:28.517+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:28.593 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:28.593+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:29.228 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:29.228+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:29.504 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:29.505+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:29.587 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:29.587+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:30.218 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:30.219+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:30.537 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:30.538+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:30.547 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:30.547+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:31.261 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:31.261+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:31.539 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:31.540+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:31.561 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:31.560+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:32.260 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:32.260+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:32.529 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:32.530+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:32.546 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:32.546+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:33.220 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:33.220+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:33.525 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:33.526+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:33.560 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:33.560+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:34.249 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:34.249+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:34.532 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:34.533+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:34.547 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:34.547+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:35.265 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:35.263+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:35.560 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:35.561+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:35.563 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:35.563+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:36.269 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:36.269+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:36.573 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:36.573+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:36.609 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:36.610+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:37.221 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:37.221+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:37.533 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:37.533+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:37.594 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:37.594+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:38.214 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:38.214+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:38.534 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:38.534+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:38.616 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:38.615+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:39.214 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:39.213+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:39.533 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:39.532+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:39.569 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:39.568+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:40.189 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:40.188+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:40.499 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:40.498+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:40.577 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:40.576+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:41.240 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:41.239+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:41.516 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:41.515+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:41.618 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:41.617+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:42.192 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:42.191+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:42.521 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:42.520+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:42.593 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:42.592+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:43.236 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:43.235+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:43.491 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:43.490+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:43.580 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:43.579+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:44.274 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:44.274+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:44.454 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:44.453+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:44.540 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:44.540+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:45.274 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:45.274+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:45.428 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:45.428+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:45.503 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:45.502+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:46.270 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:46.269+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:46.388 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:46.387+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:46.512 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:46.511+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:47.223 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:47.222+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:47.338 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:47.337+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:47.527 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:47.527+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:48.261 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:48.260+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:48.332 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:48.332+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:48.519 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:48.518+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:49.308 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:49.307+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:49.370 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:49.370+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:49.543 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:49.543+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:50.350 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:50.349+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:50.408 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:50.407+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:50.582 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:50.582+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:51.354 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:51.354+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:51.438 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:51.436+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:51.544 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:51.544+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:52.361 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:52.360+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:52.451 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:52.450+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:52.503 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:52.502+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:53.338 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:53.337+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:53.419 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:53.419+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:53.521 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:53.520+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:54.360 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:54.359+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:54.426 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:54.426+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:54.543 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:54.542+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:55.350 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:55.349+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:55.395 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:55.395+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:55.587 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:55.586+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:56.371 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:56.370+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:56.372 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:56.371+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:56.596 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:56.595+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:57.387 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:57.387+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:57.398 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:57.397+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:57.635 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:57.634+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:58.385 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:58.384+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:58.409 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:58.409+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:58.624 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:58.623+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:59.379 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:12:59.378+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:59.431 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:12:59.430+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:12:59.647 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:12:59.646+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:00.382 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:00.381+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:00.386 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:00.385+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:00.642 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:00.642+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:01.394 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:01.393+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:01.420 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:01.420+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:01.655 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:01.654+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:02.354 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:02.353+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:02.455 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:02.454+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:02.681 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:02.680+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:03.316 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:03.315+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:03.420 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:03.419+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:03.686 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:03.686+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:04.362 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:04.362+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:04.382 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:04.381+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:04.669 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:04.669+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:05.344 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:05.344+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:05.376 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:05.375+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:05.627 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:05.626+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:06.315 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:06.315+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:06.373 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:06.372+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:06.615 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:06.615+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:07.295 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:07.296+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:07.337 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:07.336+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:07.575 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:07.574+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:08.291 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:08.290+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:08.323 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:08.323+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:08.595 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:08.595+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:09.289 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:09.290+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:09.294 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:09.292+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:09.589 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:09.590+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:10.261 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:10.261+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:10.261 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:10.262+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:10.567 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:10.568+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:11.211 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:11.212+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:11.261 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:11.261+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:11.522 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:11.522+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:12.243 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:12.243+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:12.247 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:12.248+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:12.507 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:12.507+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:13.201 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:13.201+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:13.213 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:13.213+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:13.514 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:13.514+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:14.210 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:14.211+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:14.238 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:14.238+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:14.537 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:14.538+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:15.163 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:15.163+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:15.238 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:15.238+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:15.501 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:15.501+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:16.193 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:16.193+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:16.210 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:16.210+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:16.469 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:16.469+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:17.174 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:17.174+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:17.227 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:17.227+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:17.494 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:17.494+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:18.223 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:18.224+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:18.265 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:18.265+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:18.530 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:18.530+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:19.239 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:19.239+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:19.282 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:19.282+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:19.511 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:19.512+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:20.243 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:20.244+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:20.295 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:20.296+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:20.516 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:20.516+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:21.230 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:21.230+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:21.263 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:21.263+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:21.529 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:21.529+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:22.191 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:22.190+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:22.242 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:22.241+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:22.554 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:22.553+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:23.184 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:23.184+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:23.262 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:23.261+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:23.520 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:23.519+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:24.169 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:24.168+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:24.270 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:24.270+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:24.540 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:24.539+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:25.182 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:25.182+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:25.227 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:25.226+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:25.513 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:25.513+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:26.186 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:26.186+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:26.195 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:26.194+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:26.512 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:26.511+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:27.215 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:27.214+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:27.220 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:27.220+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:27.496 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:27.496+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:28.201 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:28.201+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:28.223 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:28.223+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:28.449 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:28.448+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:29.195 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:29.194+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:29.200 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:29.199+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:29.419 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:29.418+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:30.219 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:30.218+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:30.240 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:30.239+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:30.402 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:30.401+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:31.217 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:31.216+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:31.244 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:31.244+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:31.434 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:31.433+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:32.175 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:32.174+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:32.224 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:32.224+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:32.472 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:32.471+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:33.139 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:33.139+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:33.272 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:33.271+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:33.513 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:33.513+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:34.095 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:34.094+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:34.251 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:34.250+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:34.529 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:34.529+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:35.071 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:35.071+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:35.211 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:35.210+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:35.507 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:35.506+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:36.032 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:36.031+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:36.259 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:36.259+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:36.526 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:36.525+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:37.003 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:37.002+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:37.266 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:37.265+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:37.545 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:37.544+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:38.032 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:38.031+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:38.300 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:38.300+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:38.582 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:38.581+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:39.037 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:39.033+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:39.308 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:39.307+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:39.623 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:39.622+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:40.035 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:40.035+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:40.304 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:40.304+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:40.579 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:40.578+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:41.059 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:41.058+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:41.281 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:41.280+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:41.613 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:41.612+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:42.080 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:42.079+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:42.309 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:42.308+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:42.607 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:42.606+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:43.095 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:43.094+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:43.267 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:43.266+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:43.584 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:43.583+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:44.076 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:44.075+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:44.257 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:44.256+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:44.588 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:44.587+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:45.090 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:45.089+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:45.254 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:45.253+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:45.589 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:45.588+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:46.128 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:46.127+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:46.292 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:46.291+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:46.563 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:46.562+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:47.125 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:47.125+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:47.342 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:47.342+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:47.525 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:47.525+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:48.114 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:48.113+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:48.322 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:48.321+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:48.554 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:48.554+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:49.077 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:49.076+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:49.310 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:49.310+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:49.550 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:49.549+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:50.059 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:50.058+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:50.265 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:50.265+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:50.513 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:50.512+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:51.070 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:51.069+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:51.306 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:51.305+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:51.488 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:51.487+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:52.061 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:52.061+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:52.338 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:52.337+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:52.481 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:52.481+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:53.045 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:53.045+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:53.371 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:53.371+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:53.446 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:53.446+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:54.071 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:54.071+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:54.411 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:54.412+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:54.467 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:54.467+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:55.068 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:55.068+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:55.442 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:55.443+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:55.450 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:55.450+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:56.115 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:56.115+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:56.434 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:56.434+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:56.476 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:56.476+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:57.138 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:57.138+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:57.469 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:57.470+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:57.483 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:57.483+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:58.168 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:58.168+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:58.494 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:58.494+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:58.511 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:58.511+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:59.198 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:13:59.198+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:59.476 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:13:59.476+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:13:59.483 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:13:59.483+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:00.187 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:00.187+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:00.510 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:00.511+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:00.511 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:00.511+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:01.188 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:01.188+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:01.483 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:01.484+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:01.506 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:01.506+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:02.171 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:02.171+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:02.466 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:02.466+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:02.486 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:02.486+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:03.219 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:03.220+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:03.429 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:03.430+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:03.524 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:03.524+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:04.233 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:04.233+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:04.467 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:04.467+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:04.503 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:04.503+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:05.231 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:05.232+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:05.428 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:05.429+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:05.510 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:05.510+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:06.238 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:06.237+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:06.398 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:06.397+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:06.541 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:06.540+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:07.270 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:07.270+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:07.386 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:07.385+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:07.552 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:07.551+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:08.245 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:08.244+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:08.357 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:08.356+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:08.517 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:08.516+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:09.258 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:09.257+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:09.341 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:09.340+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:09.542 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:09.541+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:10.282 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:10.282+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:10.314 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:10.313+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:10.575 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:10.574+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:11.331 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:11.331+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:11.352 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:11.351+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:11.548 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:11.547+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:12.344 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:12.344+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:12.365 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:12.364+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:12.583 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:12.582+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:13.342 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:13.341+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:13.343 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:13.343+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:13.585 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:13.585+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:14.309 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:14.309+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:14.364 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:14.363+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:14.613 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:14.608+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:15.300 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:15.299+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:15.385 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:15.384+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:15.600 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:15.600+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:16.278 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:16.277+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:16.341 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:16.340+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:16.574 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:16.573+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:17.305 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:17.304+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:17.358 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:17.358+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:17.539 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:17.538+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:18.353 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:18.352+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:18.405 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:18.404+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:18.493 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:18.493+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:19.402 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:19.401+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:19.411 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:19.411+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:19.524 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:19.523+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:20.367 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:20.366+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:20.430 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:20.429+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:20.475 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:20.474+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:21.376 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:21.375+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:21.406 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:21.405+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:21.522 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:21.522+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:22.341 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:22.340+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:22.376 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:22.375+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:22.528 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:22.527+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:23.318 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:23.318+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:23.410 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:23.410+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:23.493 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:23.492+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:24.352 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:24.351+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:24.414 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:24.413+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:24.506 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:24.505+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:25.342 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:25.341+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:25.379 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:25.379+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:25.534 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:25.533+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:26.381 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:26.381+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:26.418 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:26.417+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:26.547 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:26.546+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:27.419 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:27.418+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:27.453 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:27.453+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:27.506 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:27.506+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:28.440 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:28.439+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:28.451 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:28.451+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:28.477 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:28.476+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:29.410 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:29.409+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:29.484 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:29.483+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:29.505 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:29.505+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:30.429 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:30.428+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:30.512 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:30.511+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:30.534 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:30.533+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:31.413 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:31.412+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:31.494 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:31.494+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:31.550 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:31.549+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:32.386 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:32.385+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:32.509 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:32.508+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:32.593 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:32.592+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:33.358 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:33.357+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:33.554 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:33.553+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:33.583 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:33.582+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:34.331 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:34.330+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:34.520 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:34.519+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:34.622 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:34.621+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:35.282 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:35.282+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:35.568 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:35.568+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:35.606 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:35.606+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:36.311 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:36.311+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:36.540 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:36.540+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:36.581 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:36.581+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:37.285 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:37.284+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:37.538 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:37.538+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:37.584 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:37.584+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:38.282 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:38.283+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:38.506 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:38.502+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:38.551 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:38.551+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:39.294 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:39.294+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:39.468 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:39.469+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:39.594 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:39.594+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:40.281 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:40.281+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:40.445 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:40.445+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:40.586 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:40.587+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:41.250 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:41.250+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:41.405 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:41.405+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:41.592 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:41.592+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:42.276 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:42.277+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:42.361 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:42.361+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:42.613 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:42.613+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:43.268 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:43.268+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:43.370 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:43.370+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:43.575 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:43.575+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:44.219 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:44.219+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:44.419 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:44.419+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:44.550 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:44.550+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:45.254 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:45.254+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:45.394 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:45.395+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:45.582 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:45.582+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:46.239 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:46.240+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:46.366 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:46.366+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:46.607 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:46.607+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:47.243 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:47.243+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:47.406 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:47.407+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:47.568 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:47.569+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:48.271 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:48.272+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:48.394 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:48.394+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:48.572 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:48.572+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:49.235 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:49.235+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:49.347 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:49.348+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:49.610 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:49.610+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:50.208 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:50.209+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:50.299 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:50.300+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:50.630 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:50.630+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:51.159 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:51.159+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:51.336 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:51.335+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:51.678 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:51.678+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:52.193 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:52.193+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:52.378 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:52.377+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:52.677 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:52.676+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:53.178 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:53.178+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:53.411 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:53.410+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:53.724 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:53.724+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:54.160 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:54.159+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:54.402 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:54.402+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:54.699 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:54.698+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:55.140 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:55.140+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:55.442 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:55.441+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:55.671 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:55.671+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:56.092 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:56.092+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:56.459 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:56.458+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:56.680 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:56.679+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:57.122 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:57.122+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:57.468 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:57.468+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:57.665 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:57.665+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:58.077 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:58.076+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:58.479 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:58.478+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:58.706 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:58.705+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:59.030 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:14:59.030+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:59.436 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:14:59.435+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:14:59.716 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:14:59.715+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:00.046 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:00.045+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:00.450 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:00.449+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:00.741 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:00.740+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:01.081 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:01.081+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:01.491 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:01.490+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:01.777 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:01.776+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:02.096 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:02.096+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:02.468 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:02.467+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:02.769 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:02.768+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:03.089 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:03.088+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:03.499 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:03.498+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:03.809 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:03.808+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:04.104 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:04.103+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:04.491 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:04.491+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:04.766 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:04.765+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:05.136 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:05.136+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:05.511 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:05.510+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:05.749 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:05.748+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:06.139 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:06.138+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:06.508 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:06.507+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:06.753 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:06.752+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:07.113 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:07.112+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:07.489 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:07.488+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:07.719 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:07.718+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:08.130 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:08.129+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:08.443 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:08.442+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:08.671 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:08.671+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:09.179 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:09.178+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:09.486 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:09.485+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:09.654 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:09.653+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:10.159 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:10.159+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:10.455 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:10.455+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:10.658 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:10.657+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:11.123 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:11.122+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:11.486 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:11.485+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:11.663 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:11.663+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:12.155 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:12.154+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:12.527 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:12.527+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:12.623 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:12.622+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:13.197 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:13.196+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:13.562 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:13.561+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:13.579 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:13.578+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:14.152 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:14.152+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:14.582 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:14.581+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:14.630 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:14.629+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:15.194 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:15.194+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:15.587 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:15.586+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:15.640 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:15.639+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:16.152 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:16.150+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:16.570 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:16.569+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:16.640 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:16.639+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:17.184 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:17.183+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:17.560 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:17.560+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:17.593 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:17.593+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:18.173 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:18.172+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:18.574 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:18.573+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:18.579 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:18.579+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:19.128 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:19.128+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:19.580 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:19.580+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:19.618 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:19.617+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:20.133 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:20.133+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:20.564 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:20.563+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:20.653 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:20.652+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:21.125 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:21.124+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:21.540 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:21.540+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:21.668 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:21.667+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:22.167 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:22.167+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:22.555 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:22.556+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:22.660 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:22.660+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:23.184 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:23.185+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:23.606 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:23.606+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:23.642 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:23.643+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:24.193 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:24.193+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:24.645 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:24.645+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:24.679 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:24.679+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:25.204 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:25.205+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:25.625 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:25.625+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:25.634 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:25.635+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:26.211 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:26.211+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:26.587 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:26.588+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:26.679 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:26.679+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:27.195 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:27.195+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:27.616 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:27.616+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:27.727 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:27.727+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:28.198 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:28.198+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:28.635 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:28.635+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:28.745 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:28.745+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:29.180 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:29.181+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:29.610 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:29.610+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:29.755 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:29.755+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:30.193 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:30.193+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:30.561 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:30.561+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:30.736 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:30.736+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:31.221 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:31.222+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:31.594 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:31.594+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:31.700 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:31.700+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:32.182 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:32.183+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:32.588 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:32.588+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:32.673 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:32.674+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:33.185 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:33.186+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:33.573 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:33.573+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:33.702 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:33.703+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:34.185 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:34.185+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:34.594 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:34.595+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:34.694 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:34.694+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:35.136 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:35.136+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:35.593 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:35.592+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:35.682 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:35.681+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:36.106 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:36.106+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:36.556 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:36.556+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:36.722 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:36.721+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:37.103 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:37.102+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:37.541 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:37.540+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:37.744 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:37.744+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:38.082 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:38.081+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:38.528 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:38.527+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:38.748 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:38.747+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:39.086 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:39.085+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:39.484 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:39.483+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:39.738 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:39.737+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:40.097 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:40.096+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:40.468 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:40.467+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:40.788 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:40.788+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:41.064 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:41.063+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:41.456 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:41.455+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:41.747 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:41.746+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:42.083 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:42.082+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:42.458 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:42.458+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:42.790 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:42.789+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:43.124 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:43.123+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:43.507 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:43.507+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:43.797 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:43.796+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:44.095 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:44.094+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:44.495 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:44.494+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:44.846 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:44.845+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:45.053 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:45.052+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:45.517 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:45.516+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:45.806 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:45.805+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:46.045 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:46.044+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:46.565 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:46.564+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:46.803 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:46.803+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:47.009 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:47.003+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:47.532 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:47.532+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:47.764 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:47.763+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:47.969 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:47.968+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:48.541 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:48.540+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:48.728 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:48.727+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:48.973 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:48.970+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:49.547 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:49.546+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:49.718 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:49.717+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:49.936 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:49.935+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:50.584 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:50.583+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:50.682 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:50.681+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:50.946 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:50.946+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:51.558 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:51.557+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:51.722 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:51.721+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:51.980 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:51.979+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:52.564 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:52.563+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:52.769 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:52.768+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:53.012 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:53.011+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:53.610 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:53.609+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:53.820 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:53.819+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:54.055 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:54.055+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:54.594 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:54.593+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:54.779 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:54.778+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:55.064 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:55.064+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:55.558 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:55.557+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:55.806 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:55.805+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:56.098 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:56.098+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:56.570 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:56.569+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:56.816 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:56.815+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:57.141 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:57.140+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:57.617 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:57.616+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:57.797 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:57.796+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:58.181 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:58.178+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:58.664 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:58.663+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:58.804 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:58.803+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:59.147 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:15:59.146+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:59.623 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:15:59.623+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:15:59.813 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:15:59.813+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:00.122 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:00.121+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:00.611 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:00.610+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:00.800 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:00.799+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:01.145 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:01.144+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:01.657 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:01.653+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:01.787 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:01.787+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:02.118 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:02.117+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:02.689 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:02.688+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:02.765 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:02.761+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:03.155 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:03.154+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:03.724 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:03.723+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:03.805 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:03.804+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:04.108 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:04.108+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:04.705 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:04.704+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:04.774 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:04.772+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:05.150 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:05.149+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:05.660 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:05.659+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:05.775 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:05.774+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:06.124 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:06.123+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:06.629 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:06.629+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:06.800 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:06.800+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:07.128 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:07.129+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:07.660 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:07.661+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:07.765 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:07.766+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:08.089 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:08.089+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:08.666 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:08.666+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:08.756 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:08.757+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:09.064 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:09.064+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:09.664 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:09.664+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:09.764 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:09.764+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:10.022 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:10.023+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:10.668 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:10.667+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:10.800 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:10.801+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:11.048 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:11.048+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:11.699 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:11.699+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:11.801 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:11.801+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:12.010 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:12.010+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:12.710 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:12.710+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:12.768 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:12.768+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:13.019 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:13.020+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:13.688 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:13.688+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:13.818 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:13.818+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:13.997 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:13.997+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:14.639 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:14.639+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:14.782 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:14.782+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:14.950 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:14.950+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:15.627 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:15.627+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:15.749 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:15.749+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:15.929 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:15.929+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:16.636 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:16.636+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:16.762 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:16.762+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:16.946 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:16.946+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:17.625 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:17.626+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:17.736 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:17.736+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:17.990 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:17.990+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:18.606 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:18.606+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:18.703 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:18.704+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:18.956 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:18.957+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:19.572 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:19.571+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:19.696 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:19.695+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:19.909 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:19.909+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:20.613 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:20.612+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:20.675 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:20.674+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:20.949 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:20.948+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:21.589 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:21.588+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:21.689 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:21.688+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:21.937 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:21.936+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:22.581 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:22.580+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:22.644 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:22.643+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:22.900 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:22.899+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:23.623 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:23.622+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:23.648 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:23.647+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:23.917 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:23.916+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:24.660 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:24.659+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:24.669 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:24.668+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:24.948 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:24.948+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:25.665 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:25.664+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:25.695 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:25.695+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:25.943 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:25.942+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:26.688 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:26.687+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:26.706 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:26.706+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:26.992 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:26.991+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:27.718 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:27.717+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:27.729 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:27.728+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:28.039 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:28.039+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:28.768 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:28.766+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:28.777 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:28.776+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:29.003 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:29.002+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:29.789 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:29.788+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:29.795 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:29.795+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:29.984 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:29.983+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:30.832 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:30.831+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:30.835 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:30.834+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:30.964 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:30.963+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:31.800 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:31.799+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:31.809 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:31.808+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:31.956 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:31.955+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:32.803 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:32.802+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:32.829 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:32.828+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:32.956 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:32.955+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:33.773 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:33.772+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:33.790 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:33.790+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:33.913 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:33.912+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:34.737 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:34.737+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:34.822 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:34.821+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:34.944 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:34.943+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:35.713 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:35.712+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:35.809 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:35.808+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:35.913 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:35.913+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:36.687 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:36.686+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:36.820 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:36.819+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:36.900 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:36.899+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:37.697 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:37.696+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:37.781 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:37.781+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:37.852 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:37.852+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:38.674 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:38.673+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:38.766 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:38.765+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:38.891 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:38.891+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:39.698 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:39.697+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:39.796 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:39.795+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:39.907 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:39.906+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:40.727 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:40.726+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:40.806 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:40.806+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:40.897 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:40.896+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:41.681 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:41.680+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:41.842 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:41.841+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:41.854 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:41.853+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:42.660 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:42.660+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:42.827 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:42.826+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:42.859 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:42.858+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:43.697 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:43.696+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:43.859 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:43.859+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:43.870 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:43.869+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:44.695 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:44.694+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:44.867 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:44.867+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:44.902 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:44.901+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:45.708 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:45.707+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:45.883 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:45.882+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:45.899 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:45.898+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:46.700 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:46.699+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:46.887 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:46.887+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:46.933 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:46.932+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:47.664 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:47.663+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:47.910 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:47.909+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:47.923 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:47.923+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:48.667 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:48.666+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:48.893 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:48.892+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:48.961 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:48.961+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:49.694 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:49.686+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:49.898 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:49.897+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:49.955 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:49.954+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:50.701 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:50.701+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:50.873 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:50.873+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:50.989 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:50.990+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:51.685 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:51.685+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:51.847 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:51.848+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:51.982 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:51.982+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:52.699 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:52.699+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:52.812 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:52.813+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:52.975 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:52.975+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:53.702 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:53.702+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:53.832 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:53.832+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:53.949 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:53.950+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:54.672 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:54.672+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:54.789 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:54.789+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:54.910 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:54.910+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:55.700 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:55.700+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:55.777 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:55.777+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:55.918 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:55.918+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:56.668 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:56.668+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:56.798 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:56.798+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:56.891 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:56.891+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:57.718 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:57.718+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:57.811 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:57.811+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:57.856 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:57.857+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:58.722 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:58.722+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:58.839 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:58.839+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:58.889 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:58.889+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:59.680 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:16:59.680+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:59.810 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:16:59.810+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:16:59.865 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:16:59.866+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:00.671 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:00.671+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:00.839 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:00.840+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:00.885 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:00.885+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:01.668 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:01.668+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:01.832 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:01.832+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:01.931 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:01.931+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:02.675 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:02.676+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:02.826 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:02.827+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:02.892 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:02.893+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:03.678 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:03.678+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:03.800 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:03.799+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:03.870 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:03.869+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:04.677 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:04.676+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:04.785 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:04.785+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:04.827 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:04.826+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:05.680 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:05.679+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:05.737 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:05.736+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:05.782 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:05.781+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:06.686 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:06.685+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:06.692 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:06.691+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:06.735 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:06.734+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:07.652 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:07.651+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:07.677 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:07.677+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:07.741 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:07.741+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:08.651 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:08.650+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:08.658 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:08.657+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:08.732 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:08.731+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:09.618 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:09.618+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:09.680 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:09.679+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:09.700 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:09.699+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:10.610 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:10.609+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:10.687 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:10.686+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:10.695 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:10.694+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:11.623 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:11.622+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:11.674 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:11.674+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:11.708 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:11.707+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:12.668 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:12.667+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:12.672 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:12.671+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:12.673 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:12.672+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:13.642 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:13.642+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:13.643 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:13.642+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:13.674 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:13.673+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:14.628 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:14.627+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:14.641 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:14.641+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:14.686 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:14.685+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:15.642 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:15.642+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:15.680 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:15.678+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:15.715 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:15.713+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:16.600 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:16.599+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:16.670 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:16.669+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:16.707 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:16.706+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:17.609 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:17.608+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:17.715 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:17.714+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:17.740 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:17.740+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:18.595 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:18.595+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:18.734 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:18.733+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:18.742 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:18.742+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:19.569 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:19.569+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:19.695 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:19.694+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:19.732 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:19.731+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:20.538 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:20.537+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:20.673 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:20.672+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:20.744 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:20.743+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:21.514 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:21.513+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:21.719 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:21.718+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:21.741 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:21.741+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:22.547 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:22.546+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:22.755 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:22.755+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:22.780 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:22.780+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:23.532 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:23.531+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:23.736 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:23.735+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:23.802 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:23.801+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:24.484 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:24.483+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:24.696 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:24.695+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:24.813 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:24.813+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:25.446 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:25.445+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:25.647 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:25.646+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:25.789 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:25.789+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:26.462 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:26.461+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:26.683 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:26.682+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:26.747 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:26.746+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:27.452 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:27.451+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:27.658 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:27.657+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:27.720 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:27.719+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:28.482 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:28.481+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:28.643 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:28.642+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:28.712 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:28.711+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:29.488 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:29.487+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:29.683 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:29.683+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:29.735 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:29.735+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:30.506 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:30.505+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:30.648 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:30.647+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:30.753 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:30.752+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:31.499 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:31.498+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:31.695 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:31.695+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:31.730 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:31.729+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:32.507 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:32.507+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:32.731 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:32.730+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:32.768 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:32.767+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:33.521 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:33.521+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:33.756 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:33.757+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:33.816 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:33.815+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:34.558 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:34.558+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:34.716 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:34.716+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:34.792 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:34.793+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:35.585 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:35.585+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:35.691 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:35.691+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:35.843 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:35.843+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:36.623 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:36.623+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:36.656 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:36.656+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:36.800 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:36.801+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:37.607 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:37.607+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:37.662 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:37.662+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:37.849 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:37.849+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:38.579 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:38.579+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:38.618 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:38.619+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:38.817 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:38.818+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:39.607 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:39.608+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:39.660 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:39.660+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:39.820 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:39.820+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:40.563 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:40.564+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:40.689 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:40.690+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:40.846 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:40.846+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:41.608 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:41.608+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:41.664 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:41.664+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:41.866 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:41.866+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:42.619 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:42.620+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:42.709 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:42.709+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:42.869 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:42.870+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:43.584 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:43.584+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:43.748 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:43.748+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:43.897 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:43.897+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:44.570 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:44.570+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:44.709 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:44.709+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:44.872 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:44.873+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:45.613 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:45.613+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:45.664 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:45.665+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:45.862 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:45.863+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:46.571 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:46.572+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:46.669 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:46.669+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:46.865 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:46.865+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:47.581 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:47.581+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:47.708 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:47.708+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:47.915 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:47.915+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:48.551 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:48.550+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:48.726 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:48.725+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:48.873 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:48.872+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:49.502 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:49.501+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:49.751 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:49.751+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:49.872 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:49.872+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:50.495 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:50.494+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:50.754 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:50.754+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:50.896 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:50.896+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:51.533 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:51.533+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:51.785 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:51.784+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:51.918 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:51.917+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:52.492 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:52.491+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:52.809 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:52.808+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:52.919 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:52.918+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:53.474 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:53.473+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:53.773 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:53.773+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:53.922 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:53.921+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:54.435 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:54.434+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:54.772 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:54.772+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:54.926 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:54.925+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:55.449 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:55.449+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:55.756 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:55.756+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:55.924 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:55.924+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:56.478 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:56.477+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:56.804 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:56.803+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:56.876 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:56.875+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:57.465 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:57.464+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:57.799 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:57.798+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:57.861 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:57.860+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:58.493 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:58.493+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:58.815 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:58.814+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:58.849 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:58.846+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:59.518 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:17:59.517+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:59.822 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:17:59.821+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:17:59.859 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:17:59.859+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:00.498 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:00.497+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:00.817 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:00.816+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:00.830 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:00.829+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:01.498 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:01.497+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:01.796 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:01.795+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:01.847 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:01.846+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:02.494 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:02.493+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:02.752 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:02.751+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:02.870 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:02.870+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:03.529 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:03.528+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:03.761 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:03.760+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:03.868 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:03.868+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:04.547 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:04.546+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:04.737 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:04.736+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:04.869 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:04.868+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:05.527 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:05.526+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:05.747 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:05.747+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:05.830 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:05.829+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:06.523 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:06.522+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:06.793 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:06.792+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:06.813 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:06.813+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:07.568 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:07.567+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:07.762 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:07.762+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:07.765 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:07.764+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:08.531 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:08.531+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:08.781 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:08.779+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:08.781 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:08.779+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:09.563 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:09.562+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:09.769 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:09.769+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:09.781 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:09.781+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:10.573 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:10.572+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:10.733 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:10.733+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:10.742 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:10.742+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:11.566 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:11.565+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:11.685 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:11.684+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:11.752 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:11.751+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:12.585 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:12.584+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:12.728 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:12.728+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:12.734 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:12.733+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:13.583 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:13.583+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:13.756 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:13.755+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:13.778 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:13.777+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:14.581 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:14.581+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:14.759 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:14.759+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:14.769 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:14.768+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:15.533 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:15.533+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:15.725 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:15.724+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:15.747 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:15.746+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:16.494 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:16.493+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:16.698 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:16.697+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:16.775 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:16.774+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:17.457 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:17.457+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:17.736 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:17.736+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:17.778 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:17.777+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:18.492 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:18.490+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:18.755 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:18.754+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:18.822 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:18.821+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:19.488 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:19.488+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:19.769 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:19.769+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:19.776 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:19.776+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:20.464 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:20.464+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:20.765 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:20.765+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:20.796 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:20.796+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:21.473 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:21.473+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:21.734 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:21.734+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:21.790 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:21.790+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:22.519 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:22.519+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:22.756 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:22.757+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:22.768 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:22.768+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:23.514 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:23.514+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:23.755 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:23.756+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:23.794 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:23.794+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:24.506 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:24.507+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:24.776 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:24.776+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:24.829 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:24.830+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:25.541 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:25.541+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:25.797 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:25.797+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:25.799 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:25.799+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:26.586 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:26.586+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:26.752 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:26.752+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:26.820 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:26.820+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:27.583 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:27.583+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:27.800 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:27.801+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:27.833 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:27.833+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:28.568 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:28.568+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:28.801 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:28.801+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:28.825 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:28.825+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:29.525 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:29.525+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:29.781 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:29.781+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:29.819 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:29.819+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:30.516 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:30.516+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:30.740 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:30.740+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:30.792 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:30.792+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:31.525 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:31.525+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:31.746 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:31.746+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:31.752 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:31.752+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:32.551 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:32.550+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:32.715 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:32.714+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:32.787 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:32.786+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:33.600 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:33.599+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:34.369 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:33.691+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:34.370 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:33.781+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:34.574 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:34.573+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:34.665 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:34.664+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:34.800 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:34.799+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:35.559 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:35.557+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:35.661 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:35.659+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:35.822 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:35.821+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:36.521 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:36.520+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:36.671 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:36.670+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:36.858 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:36.857+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:37.555 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:37.554+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:37.716 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:37.715+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:37.832 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:37.831+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:38.524 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:38.523+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:38.764 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:38.763+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:38.839 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:38.838+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:39.508 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:39.507+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:39.725 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:39.724+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:39.825 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:39.824+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:40.489 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:40.488+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:40.751 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:40.750+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:40.799 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:40.798+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:41.516 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:41.515+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:41.755 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:41.754+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:41.783 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:41.783+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:42.528 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:42.527+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:42.736 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:42.735+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:42.745 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:42.744+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:43.574 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:43.573+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:43.708 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:43.707+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:43.732 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:43.731+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:44.610 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:44.610+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:44.736 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:44.735+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:44.774 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:44.774+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:45.624 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:45.624+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:45.756 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:45.754+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:45.789 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:45.788+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:46.576 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:46.576+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:46.789 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:46.786+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:46.805 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:46.804+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:47.611 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:47.610+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:47.815 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:47.815+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:47.824 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:47.824+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:48.596 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:48.595+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:48.799 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:48.799+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:48.865 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:48.864+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:49.619 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:49.619+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:49.768 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:49.767+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:49.875 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:49.874+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:50.622 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:50.621+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:50.780 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:50.779+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:50.848 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:50.848+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:51.652 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:51.652+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:51.758 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:51.757+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:51.866 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:51.866+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:52.614 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:52.614+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:52.738 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:52.737+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:52.859 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:52.859+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:53.589 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:53.588+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:53.703 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:53.702+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:53.871 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:53.870+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:54.596 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:54.595+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:54.732 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:54.732+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:54.845 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:54.844+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:55.599 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:55.598+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:55.782 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:55.782+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:55.882 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:55.882+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:56.560 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:56.560+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:56.765 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:56.764+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:56.840 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:56.840+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:57.580 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:57.580+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:57.755 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:57.754+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:57.828 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:57.827+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:58.566 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:58.566+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:58.716 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:58.716+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:58.839 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:58.838+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:59.590 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:18:59.589+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:59.725 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:18:59.724+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:18:59.864 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:18:59.863+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:00.604 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:00.603+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:00.693 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:00.693+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:00.877 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:00.876+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:01.576 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:01.576+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:01.683 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:01.682+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:01.872 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:01.871+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:02.591 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:02.591+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:02.673 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:02.673+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:02.884 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:02.883+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:03.553 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:03.552+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:03.700 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:03.700+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:03.876 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:03.876+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:04.517 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:04.516+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:04.739 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:04.739+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:04.851 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:04.851+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:05.537 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:05.538+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:05.763 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:05.763+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:05.822 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:05.822+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:06.548 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:06.548+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:06.784 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:06.784+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:06.788 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:06.788+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:07.501 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:07.502+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:07.765 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:07.765+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:07.806 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:07.806+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:08.493 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:08.493+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:08.801 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:08.801+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:08.836 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:08.836+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:09.542 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:09.542+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:09.790 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:09.791+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:09.862 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:09.863+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:10.537 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:10.537+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:10.783 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:10.784+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:10.848 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:10.848+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:11.527 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:11.527+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:11.814 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:11.814+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:11.855 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:11.855+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:12.528 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:12.528+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:12.771 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:12.771+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:12.826 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:12.826+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:13.540 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:13.541+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:13.737 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:13.738+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:13.832 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:13.832+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:14.565 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:14.565+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:14.748 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:14.748+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:14.866 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:14.866+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:15.547 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:15.547+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:15.785 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:15.785+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:15.820 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:15.821+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:16.558 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:16.558+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:16.765 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:16.765+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:16.826 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:16.826+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:17.561 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:17.560+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:17.750 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:17.749+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:17.798 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:17.797+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:18.599 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:18.598+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:18.776 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:18.775+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:18.796 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:18.795+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:19.640 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:19.640+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:19.752 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:19.751+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:19.798 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:19.797+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:20.617 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:20.616+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:20.712 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:20.712+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:20.789 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:20.788+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:21.608 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:21.607+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:21.675 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:21.675+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:21.832 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:21.831+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:22.617 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:22.616+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:22.664 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:22.663+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:22.826 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:22.825+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:23.631 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:23.631+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:23.651 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:23.650+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:23.869 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:23.868+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:24.615 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:24.615+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:24.657 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:24.656+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:24.863 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:24.863+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:25.648 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:25.647+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:25.662 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:25.660+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:25.816 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:25.815+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:26.682 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:26.681+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:26.684 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:26.683+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:26.837 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:26.836+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:27.689 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:27.689+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:27.699 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:27.698+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:27.808 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:27.807+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:28.662 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:28.661+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:28.670 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:28.670+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:28.848 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:28.847+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:29.645 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:29.644+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:29.653 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:29.653+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:29.846 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:29.845+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:30.644 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:30.644+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:30.650 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:30.649+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:30.865 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:30.864+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:31.633 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:31.632+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:31.664 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:31.663+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:31.834 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:31.833+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:32.594 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:32.593+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:32.693 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:32.692+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:32.798 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:32.797+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:33.642 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:33.641+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:33.741 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:33.740+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:33.827 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:33.827+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:34.634 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:34.634+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:34.695 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:34.694+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:34.809 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:34.809+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:35.631 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:35.630+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:35.733 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:35.732+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:35.773 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:35.773+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:36.673 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:36.672+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:36.781 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:36.780+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:36.814 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:36.813+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:37.649 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:37.648+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:37.797 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:37.796+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:37.806 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:37.806+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:38.629 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:38.628+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:38.799 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:38.799+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:38.808 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:38.807+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:39.664 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:39.664+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:39.805 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:39.804+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:39.805 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:39.804+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:40.714 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:40.713+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:40.784 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:40.783+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:40.830 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:40.829+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:41.707 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:41.706+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:41.753 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:41.752+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:41.858 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:41.856+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:42.737 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:42.736+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:42.797 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:42.797+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:42.865 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:42.864+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:43.773 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:43.772+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:43.820 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:43.819+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:43.836 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:43.836+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:44.723 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:44.722+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 18 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:44.785 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:44.785+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:44.857 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:44.857+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:45.713 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:45.713+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 17 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:45.745 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:45.744+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:45.842 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:45.841+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:46.684 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:46.683+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 17 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:46.780 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:46.780+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:46.849 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:46.848+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:47.715 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:47.716+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 17 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:47.812 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:47.811+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:47.813 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:47.813+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:48.675 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:48.676+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 17 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:48.767 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:48.767+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:48.771 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:48.772+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:49.683 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:49.683+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 17 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:49.755 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:49.755+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:49.787 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:49.787+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:50.719 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:50.719+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 17 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:50.757 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:50.758+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:50.776 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:50.776+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:51.753 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:51.753+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 17 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:51.791 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:51.790+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:51.818 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:51.818+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:52.771 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:52.771+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:52.794 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:52.794+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 17 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:52.858 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:52.858+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:53.788 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:53.788+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 17 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:53.798 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:53.799+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:53.809 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:53.810+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:54.763 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:54.764+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:54.791 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:54.791+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:54.803 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:54.804+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 17 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:55.777 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:55.778+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 17 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:55.783 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:55.784+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:55.797 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:55.797+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:56.784 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:56.784+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:56.808 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:56.809+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 17 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:56.830 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:56.830+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:57.802 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:57.802+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 17 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:57.815 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:57.816+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:57.819 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:57.820+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:58.772 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:58.772+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:58.775 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:58.775+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:58.838 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:58.838+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 17 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:59.727 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:19:59.727+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:59.798 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:19:59.798+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:19:59.825 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:19:59.826+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 17 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:00.745 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:00.746+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:00.770 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:00.770+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:00.781 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:00.782+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 17 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:01.783 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:01.782+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:01.810 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:01.809+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 17 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:01.813 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:01.813+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:02.749 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:02.745+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:02.812 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:02.811+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 17 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:02.817 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:02.816+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:03.788 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:03.788+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:03.789 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:03.788+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 17 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:03.828 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:03.827+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:04.768 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:04.767+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:04.783 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:04.782+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 17 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:04.857 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:04.857+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:05.760 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:05.760+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:05.776 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:05.775+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 17 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:05.828 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:05.827+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:06.729 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:06.729+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 17 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:06.736 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:06.736+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:06.811 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:06.810+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:07.715 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:07.714+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 17 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:07.763 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:07.762+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:07.821 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:07.820+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:08.693 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:08.693+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 17 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:08.809 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:08.808+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:08.816 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:08.815+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:09.669 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:09.669+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 17 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:09.787 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:09.787+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:09.808 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:09.807+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:10.633 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:10.632+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 17 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:10.819 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:10.818+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:10.857 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:10.856+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:11.675 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:11.674+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 17 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:11.830 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:11.829+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:11.836 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:11.836+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:12.691 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:12.691+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 17 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:12.823 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:12.822+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:12.863 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:12.863+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:13.725 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:13.724+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 17 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:13.821 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:13.821+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:13.843 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:13.842+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:14.735 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:14.734+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 17 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:14.833 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:14.832+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:14.840 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:14.840+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:15.760 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:15.759+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:15.834 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:15.833+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:15.886 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:15.885+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:16.769 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:16.768+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:16.805 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:16.804+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:16.853 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:16.853+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:17.804 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:17.804+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:17.843 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:17.842+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:17.870 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:17.870+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:18.772 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:18.772+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:18.830 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:18.829+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:18.884 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:18.883+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:19.723 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:19.723+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:19.876 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:19.875+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:19.914 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:19.913+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:20.729 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:20.728+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:20.843 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:20.842+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:20.891 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:20.890+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:21.731 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:21.730+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:21.850 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:21.849+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:21.887 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:21.887+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:22.726 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:22.725+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:22.834 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:22.833+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:22.928 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:22.927+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:23.678 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:23.678+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:23.855 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:23.854+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:23.957 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:23.957+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:24.665 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:24.664+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:24.885 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:24.885+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:24.945 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:24.945+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:25.672 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:25.671+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:25.844 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:25.843+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:25.941 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:25.941+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:26.681 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:26.680+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:26.875 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:26.874+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:26.925 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:26.925+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:27.685 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:27.684+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:27.882 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:27.881+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:27.896 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:27.895+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:28.735 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:28.734+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:28.842 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:28.841+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:28.916 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:28.915+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:29.693 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:29.692+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:29.883 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:29.882+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:29.884 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:29.883+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:30.686 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:30.686+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:30.879 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:30.879+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:30.885 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:30.885+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:31.654 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:31.653+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:31.868 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:31.867+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:31.883 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:31.883+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:32.631 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:32.632+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:32.838 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:32.838+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:32.839 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:32.839+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:33.596 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:33.596+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:33.791 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:33.791+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:33.818 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:33.818+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:34.556 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:34.556+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:34.808 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:34.808+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:34.854 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:34.854+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:35.551 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:35.551+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:35.776 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:35.777+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:35.883 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:35.883+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:36.592 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:36.592+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:36.774 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:36.774+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:36.840 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:36.841+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:37.636 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:37.636+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:37.754 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:37.755+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:37.851 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:37.851+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:38.616 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:38.617+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:38.713 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:38.714+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:38.869 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:38.869+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:39.594 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:39.594+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:39.708 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:39.708+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:39.836 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:39.836+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:40.551 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:40.552+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:40.661 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:40.662+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:40.877 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:40.878+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:41.562 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:41.562+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:41.633 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:41.634+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:41.926 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:41.926+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:42.566 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:42.566+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:42.589 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:42.590+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:42.958 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:42.958+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:43.565 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:43.566+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:43.598 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:43.599+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:43.924 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:43.924+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:44.543 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:44.543+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:44.595 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:44.595+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:44.882 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:44.882+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:45.589 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:45.589+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:45.615 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:45.615+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:45.868 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:45.867+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:46.627 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:46.626+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:46.631 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:46.630+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:46.917 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:46.917+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:47.606 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:47.605+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:47.664 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:47.663+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:47.949 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:47.948+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:48.626 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:48.626+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:48.659 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:48.659+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:48.988 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:48.986+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:49.637 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:49.636+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:49.702 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:49.701+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:49.938 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:49.937+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:50.643 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:50.643+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:50.723 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:50.722+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:50.946 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:50.945+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:51.619 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:51.618+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:51.721 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:51.720+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:51.934 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:51.933+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:52.582 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:52.581+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:52.676 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:52.675+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:52.922 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:52.921+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:53.578 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:53.578+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:53.638 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:53.637+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:53.880 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:53.879+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:54.535 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:54.534+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:54.656 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:54.656+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:54.895 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:54.894+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:55.531 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:55.530+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:55.642 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:55.641+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:55.937 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:55.937+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:56.524 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:56.524+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:56.635 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:56.635+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:56.957 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:56.956+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:57.512 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:57.511+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:57.666 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:57.666+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:57.953 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:57.953+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:58.506 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:58.505+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:58.689 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:58.689+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:58.988 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:58.987+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:59.489 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:20:59.488+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:59.733 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:20:59.732+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:20:59.982 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:20:59.981+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:00.482 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:00.482+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:00.748 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:00.747+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:00.985 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:00.985+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:01.459 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:01.458+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:01.725 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:01.724+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:01.940 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:01.939+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:02.502 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:02.499+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:02.712 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:02.711+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:02.944 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:02.944+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:03.520 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:03.519+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:03.677 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:03.677+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:03.984 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:03.983+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:04.502 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:04.501+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:04.676 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:04.675+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:04.956 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:04.954+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:05.544 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:05.543+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:05.674 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:05.673+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:05.922 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:05.921+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:06.576 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:06.575+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:06.671 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:06.670+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:06.951 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:06.950+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:07.623 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:07.622+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:07.639 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:07.638+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:07.911 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:07.910+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:08.635 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:08.634+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:08.675 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:08.675+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:08.864 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:08.863+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:09.652 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:09.651+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 20 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:09.681 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:09.681+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:09.826 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:09.825+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:10.681 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:10.680+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:10.691 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:10.690+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:10.852 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:10.851+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:11.692 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:11.691+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:11.725 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:11.724+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:11.865 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:11.864+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:12.701 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:12.701+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:12.745 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:12.744+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:12.822 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:12.822+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:13.678 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:13.677+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:13.731 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:13.730+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:13.806 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:13.805+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:14.691 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:14.690+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:14.698 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:14.698+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:14.844 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:14.843+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:15.643 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:15.643+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:15.740 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:15.739+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:15.862 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:15.860+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:16.682 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:16.683+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:16.764 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:16.764+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:16.843 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:16.842+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:17.670 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:17.670+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:17.740 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:17.741+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:17.859 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:17.859+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:18.697 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:18.698+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:18.791 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:18.791+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:18.828 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:18.828+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:19.745 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:19.746+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:19.785 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:19.785+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:19.829 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:19.829+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:20.751 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:20.751+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:20.771 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:20.771+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:20.822 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:20.822+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:21.776 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:21.777+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:21.779 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:21.779+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:21.814 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:21.814+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:22.736 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:22.736+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:22.789 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:22.787+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:22.802 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:22.803+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:23.756 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:23.756+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:23.783 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:23.783+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:23.793 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:23.794+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:24.736 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:24.736+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:24.771 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:24.771+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:24.784 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:24.784+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:25.688 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:25.689+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:25.739 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:25.739+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:25.745 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:25.745+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:26.701 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:26.701+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:26.708 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:26.709+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:26.722 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:26.722+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:27.670 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:27.670+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:27.672 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:27.673+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:27.706 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:27.706+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:28.661 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:28.661+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:28.675 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:28.676+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:28.686 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:28.687+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:29.613 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:29.613+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:29.739 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:29.663+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:29.739 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:29.680+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:30.615 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:30.615+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:30.678 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:30.678+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:30.685 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:30.685+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:31.640 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:31.639+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:31.642 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:31.641+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:31.721 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:31.720+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:32.621 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:32.620+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:32.634 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:32.633+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:32.724 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:32.723+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:33.602 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:33.602+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:33.606 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:33.605+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:33.713 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:33.712+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:34.648 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:34.647+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:34.650 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:34.650+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:34.738 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:34.737+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:35.606 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:35.605+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:35.660 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:35.659+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:35.788 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:35.787+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:36.601 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:36.601+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:36.679 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:36.678+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:36.789 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:36.788+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:37.561 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:37.557+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:37.710 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:37.710+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:37.796 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:37.794+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:38.510 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:38.509+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:38.687 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:38.686+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:38.761 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:38.760+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:39.523 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:39.523+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:39.691 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:39.690+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:39.754 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:39.754+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:40.511 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:40.511+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:40.647 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:40.646+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:40.733 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:40.732+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:41.496 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:41.495+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:41.652 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:41.651+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:41.763 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:41.763+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:42.520 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:42.519+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:42.642 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:42.642+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:42.797 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:42.797+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:43.520 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:43.520+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:43.650 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:43.649+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:43.765 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:43.764+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:44.567 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:44.566+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:44.680 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:44.679+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:44.815 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:44.814+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:45.607 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:45.606+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:45.702 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:45.701+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:45.854 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:45.853+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:46.573 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:46.572+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:46.704 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:46.704+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:46.862 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:46.862+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:47.604 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:47.603+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:47.662 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:47.661+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:47.861 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:47.860+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:48.617 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:48.616+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:48.652 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:48.652+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:48.839 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:48.838+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:49.595 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:49.594+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:49.678 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:49.677+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:49.842 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:49.841+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:50.631 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:50.630+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:50.650 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:50.649+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:50.863 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:50.862+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:51.607 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:51.606+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:51.675 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:51.674+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:51.836 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:51.836+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:52.636 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:52.635+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:52.645 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:52.644+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:52.828 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:52.827+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:53.622 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:53.621+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:53.664 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:53.663+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:53.877 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:53.876+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:54.634 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:54.633+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:54.636 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:54.636+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:54.845 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:54.844+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:55.611 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:55.610+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:55.613 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:55.613+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:55.874 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:55.873+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:56.618 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:56.617+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:56.638 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:56.638+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:56.884 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:56.883+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:57.619 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:57.619+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:57.663 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:57.662+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:57.889 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:57.888+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:58.629 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:58.628+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:58.686 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:58.686+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:58.896 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:58.895+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:59.634 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:21:59.634+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:59.664 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:21:59.664+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:21:59.898 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:21:59.898+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:00.644 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:00.644+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:00.699 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:00.698+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:00.911 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:00.910+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:01.676 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:01.676+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:01.695 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:01.696+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:01.907 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:01.907+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:02.696 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:02.696+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:02.742 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:02.742+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:02.862 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:02.863+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:03.684 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:03.684+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:03.776 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:03.776+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:03.856 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:03.857+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:04.671 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:04.671+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:04.748 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:04.748+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:04.882 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:04.881+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:05.666 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:05.666+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:05.768 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:05.768+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:05.852 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:05.852+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:06.651 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:06.651+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:06.753 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:06.753+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:06.850 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:06.850+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:07.669 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:07.669+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:07.797 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:07.797+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:07.841 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:07.841+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:08.653 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:08.653+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:08.799 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:08.799+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:08.832 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:08.832+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:09.656 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:09.656+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:09.787 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:09.787+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:09.855 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:09.856+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:10.672 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:10.672+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:10.757 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:10.757+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:10.889 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:10.889+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:11.706 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:11.706+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:11.757 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:11.757+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:11.869 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:11.869+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:12.716 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:12.716+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:12.742 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:12.743+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:12.911 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:12.911+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:13.735 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:13.735+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:13.737 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:13.737+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:13.953 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:13.954+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:14.713 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:14.713+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:14.773 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:14.772+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:14.907 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:14.907+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:15.696 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:15.696+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:15.739 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:15.738+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:15.944 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:15.943+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:16.692 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:16.691+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:16.717 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:16.716+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:16.922 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:16.921+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:17.730 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:17.729+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:17.737 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:17.736+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:17.881 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:17.880+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:18.723 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:18.722+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:18.758 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:18.757+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:18.906 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:18.905+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:19.690 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:19.690+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:19.726 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:19.725+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:19.883 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:19.882+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:20.688 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:20.686+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:20.688 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:20.688+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:20.914 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:20.913+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:21.694 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:21.694+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:21.713 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:21.712+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:21.902 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:21.902+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:22.692 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:22.692+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:22.698 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:22.697+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:22.863 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:22.863+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:23.703 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:23.702+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:23.725 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:23.724+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:23.858 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:23.858+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:24.672 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:24.672+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:24.760 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:24.759+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:24.899 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:24.898+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:25.698 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:25.697+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:25.810 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:25.809+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:25.935 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:25.934+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:26.692 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:26.691+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:26.816 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:26.816+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:26.919 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:26.919+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:27.699 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:27.698+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:27.820 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:27.820+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:27.926 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:27.925+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:28.704 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:28.703+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:28.784 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:28.784+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:28.947 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:28.946+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:29.669 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:29.668+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:29.796 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:29.796+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:29.953 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:29.953+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:30.710 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:30.709+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:30.833 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:30.833+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:30.918 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:30.917+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:31.696 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:31.695+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:31.838 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:31.838+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:31.916 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:31.915+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:32.727 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:32.727+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:32.837 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:32.836+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:32.922 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:32.921+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:33.736 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:33.735+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:33.855 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:33.854+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:33.971 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:33.970+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:34.763 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:34.763+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:34.876 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:34.876+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:34.981 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:34.980+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:35.738 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:35.737+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:35.847 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:35.847+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:36.012 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:36.011+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:36.762 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:36.761+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:36.815 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:36.814+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:37.049 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:37.048+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:37.760 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:37.759+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:37.784 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:37.783+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:38.018 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:38.017+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:38.774 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:38.773+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:38.806 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:38.805+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:39.057 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:39.055+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:39.789 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:39.789+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:39.840 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:39.839+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:40.046 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:40.045+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:40.765 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:40.764+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:40.890 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:40.889+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:41.062 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:41.062+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:41.723 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:41.722+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:41.889 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:41.888+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:42.086 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:42.085+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:42.744 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:42.744+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:42.857 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:42.856+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:43.105 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:43.104+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:43.765 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:43.764+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:43.902 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:43.901+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:44.111 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:44.110+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:44.730 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:44.729+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:44.869 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:44.868+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:45.156 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:45.156+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:45.721 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:45.721+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:45.881 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:45.881+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:46.110 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:46.110+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:46.742 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:46.742+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:46.870 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:46.870+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:47.064 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:47.063+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:47.767 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:47.767+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:47.846 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:47.847+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:48.076 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:48.076+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:48.796 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:48.796+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:48.830 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:48.830+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:49.066 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:49.066+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:49.749 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:49.749+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:49.852 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:49.853+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:50.048 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:50.048+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:50.771 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:50.771+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:50.863 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:50.863+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:51.059 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:51.060+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:51.770 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:51.771+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:51.909 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:51.909+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:52.019 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:52.019+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:52.793 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:52.793+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:52.955 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:52.955+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:52.995 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:52.995+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:53.822 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:53.823+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:54.001 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:54.001+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:54.041 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:54.041+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:54.846 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:54.846+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:54.978 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:54.978+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:55.030 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:55.031+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:55.855 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:55.856+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:55.996 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:55.995+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:56.038 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:56.038+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:56.900 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:56.901+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:56.986 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:56.986+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:57.081 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:57.081+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:57.887 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:57.887+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:57.937 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:57.938+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:58.041 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:58.041+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:58.930 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:58.929+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:58.935 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:58.934+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:59.064 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:22:59.063+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:59.899 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:22:59.898+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:22:59.969 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:22:59.968+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:00.092 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:00.091+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:00.890 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:00.890+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:00.930 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:00.929+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:01.109 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:01.108+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:01.859 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:01.859+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:01.975 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:01.974+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:02.068 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:02.068+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:02.866 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:02.865+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:02.978 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:02.976+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:03.076 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:03.076+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:03.858 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:03.858+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:03.950 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:03.950+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:04.103 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:04.102+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:04.906 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:04.905+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:04.921 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:04.920+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:05.103 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:05.102+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:05.868 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:05.867+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:05.960 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:05.960+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:06.103 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:06.102+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:06.824 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:06.823+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:06.972 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:06.971+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:07.086 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:07.085+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:07.824 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:07.822+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:07.940 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:07.940+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:08.077 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:08.073+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:08.853 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:08.852+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:08.928 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:08.924+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:09.054 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:09.053+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:09.864 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:09.863+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:09.970 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:09.969+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:10.053 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:10.052+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:10.906 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:10.905+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:10.999 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:10.998+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:11.102 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:11.101+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:11.868 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:11.867+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:11.955 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:11.954+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:12.110 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:12.109+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:12.856 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:12.855+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:12.995 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:12.994+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:13.123 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:13.122+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:13.856 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:13.855+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:14.004 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:14.004+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:14.084 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:14.083+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:14.828 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:14.828+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:14.977 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:14.977+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:15.095 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:15.095+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:15.784 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:15.783+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:15.990 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:15.986+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:16.143 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:16.142+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:16.821 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:16.821+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:16.947 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:16.947+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:17.161 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:17.160+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:17.821 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:17.820+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:17.949 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:17.948+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:18.203 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:18.202+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:18.816 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:18.815+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:18.954 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:18.953+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:19.183 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:19.182+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:19.796 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:19.795+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:19.933 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:19.932+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:20.171 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:20.170+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:20.823 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:20.822+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:20.947 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:20.946+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:21.161 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:21.160+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:21.835 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:21.835+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:21.953 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:21.952+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:22.140 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:22.139+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:22.812 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:22.808+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:22.908 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:22.907+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:23.171 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:23.170+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:23.783 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:23.782+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:23.941 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:23.940+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:24.185 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:24.184+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:24.758 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:24.757+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:24.910 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:24.909+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:25.208 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:25.208+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:25.805 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:25.804+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:25.900 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:25.899+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:26.221 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:26.220+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:26.830 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:26.829+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:26.859 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:26.854+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:27.207 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:27.206+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:27.855 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:27.854+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:27.889 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:27.888+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:28.257 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:28.256+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:28.810 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:28.810+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:28.927 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:28.926+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:29.260 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:29.260+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:29.847 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:29.847+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:29.941 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:29.940+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:30.283 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:30.283+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:30.833 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:30.834+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:30.978 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:30.978+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:31.263 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:31.263+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:31.846 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:31.846+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:32.011 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:32.011+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:32.248 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:32.249+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:32.888 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:32.888+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:32.971 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:32.971+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:33.231 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:33.231+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:33.846 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:33.846+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:34.009 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:34.009+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:34.261 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:34.261+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:34.855 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:34.855+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:34.994 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:34.994+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:35.286 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:35.286+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:35.823 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:35.824+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:36.010 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:36.010+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:36.298 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:36.299+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:36.827 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:36.827+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:36.985 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:36.985+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:37.300 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:37.300+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:37.814 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:37.814+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:38.036 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:38.036+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:38.329 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:38.329+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:38.799 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:38.800+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:39.012 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:39.013+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:39.359 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:39.359+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:39.763 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:39.764+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:40.002 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:40.003+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:40.335 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:40.335+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:40.804 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:40.804+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:40.979 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:40.979+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:41.288 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:41.288+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:41.825 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:41.825+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:42.002 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:42.002+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:42.297 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:42.297+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:42.820 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:42.820+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:43.048 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:43.049+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:43.324 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:43.323+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:43.833 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:43.833+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:44.011 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:44.010+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:44.344 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:44.343+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:44.858 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:44.857+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:45.052 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:45.051+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:45.394 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:45.393+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:45.812 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:45.812+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:46.030 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:46.030+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:46.423 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:46.422+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:46.783 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:46.782+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:47.052 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:47.051+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:47.408 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:47.408+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:47.821 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:47.820+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:48.068 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:48.068+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:48.363 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:48.362+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:48.779 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:48.779+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:49.049 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:49.049+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:49.339 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:49.338+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:49.824 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:49.823+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:50.040 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:50.040+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:50.333 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:50.332+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:50.797 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:50.797+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:51.083 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:51.082+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:51.377 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:51.376+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:51.810 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:51.809+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:52.123 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:52.122+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:52.347 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:52.346+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:52.814 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:52.813+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:53.075 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:53.074+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:53.342 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:53.341+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:53.855 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:53.855+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:54.084 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:54.084+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:54.333 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:54.332+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:54.858 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:54.857+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:55.121 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:55.120+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:55.340 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:55.339+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:55.825 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:55.824+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:56.082 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:56.081+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:56.379 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:56.378+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:56.845 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:56.845+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:57.033 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:57.032+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:57.387 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:57.386+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:57.800 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:57.799+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:58.072 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:58.071+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:58.422 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:58.422+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:58.759 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:58.758+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:59.030 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:59.029+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:59.400 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:23:59.399+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:59.783 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:23:59.783+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:23:59.984 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:23:59.983+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:00.429 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:00.428+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:00.813 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:00.813+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:00.944 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:00.943+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:01.387 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:01.386+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:01.797 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:01.795+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:01.956 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:01.956+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:02.378 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:02.378+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:02.793 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:02.792+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:03.004 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:03.004+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:03.388 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:03.388+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:03.783 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:03.782+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:04.027 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:04.026+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:04.391 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:04.390+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:04.778 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:04.778+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:05.062 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:05.061+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:05.391 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:05.391+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:05.760 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:05.760+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:06.072 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:06.072+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:06.427 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:06.426+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:06.804 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:06.804+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:07.059 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:07.058+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:07.469 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:07.468+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:07.770 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:07.770+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:08.028 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:08.027+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:08.434 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:08.434+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:08.807 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:08.807+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:09.023 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:09.022+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:09.385 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:09.385+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:09.794 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:09.793+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:10.050 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:10.049+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:10.431 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:10.430+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:10.808 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:10.807+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:11.058 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:11.057+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:11.401 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:11.400+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:11.813 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:11.812+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:12.107 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:12.107+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:12.437 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:12.436+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:12.818 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:12.818+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:13.090 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:13.089+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:13.404 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:13.403+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:13.808 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:13.808+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:14.115 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:14.114+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:14.369 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:14.368+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:14.846 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:14.847+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:15.142 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:15.142+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:15.402 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:15.402+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:15.854 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:15.854+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:16.149 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:16.149+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:16.450 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:16.450+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:16.874 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:16.875+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:17.168 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:17.166+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:17.490 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:17.490+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:17.902 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:17.902+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:18.174 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:18.174+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:18.534 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:18.534+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:18.890 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:18.890+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:19.163 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:19.163+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:19.543 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:19.543+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:19.891 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:19.891+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:20.162 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:20.162+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:20.578 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:20.578+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:20.859 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:20.859+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:21.173 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:21.173+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:21.581 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:21.581+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:21.904 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:21.904+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:22.183 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:22.184+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:22.551 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:22.552+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:22.931 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:22.931+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:23.137 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:23.138+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:23.505 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:23.506+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:23.967 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:23.967+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:24.161 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:24.161+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:24.503 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:24.503+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:24.937 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:24.937+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:25.148 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:25.148+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:25.522 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:25.522+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:25.967 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:25.967+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:26.173 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:26.172+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:26.473 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:26.474+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:26.935 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:26.935+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:27.123 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:27.123+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:27.437 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:27.437+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:27.925 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:27.924+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:28.084 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:28.083+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:28.476 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:28.476+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:28.905 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:28.905+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:29.114 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:29.113+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:29.506 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:29.505+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:29.947 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:29.947+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:30.117 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:30.117+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:30.497 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:30.496+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:30.916 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:30.915+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:31.126 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:31.125+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:31.532 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:31.531+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:31.899 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:31.898+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:32.167 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:32.167+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:32.519 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:32.518+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:32.913 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:32.912+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:33.196 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:33.195+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:33.473 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:33.472+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:33.891 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:33.890+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:34.210 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:34.209+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:34.493 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:34.492+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:34.931 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:34.930+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:35.167 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:35.166+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:35.479 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:35.478+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:35.899 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:35.898+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:36.148 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:36.147+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:36.485 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:36.484+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:36.903 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:36.902+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:37.147 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:37.146+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:37.473 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:37.471+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:37.951 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:37.950+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:38.181 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:38.181+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:38.520 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:38.519+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:38.942 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:38.941+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:39.226 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:39.225+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:39.531 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:39.531+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:39.947 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:39.946+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:40.180 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:40.179+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:40.520 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:40.519+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:40.956 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:40.956+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:41.161 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:41.159+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:41.478 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:41.477+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:42.002 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:42.001+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:42.174 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:42.173+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:42.463 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:42.463+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:43.015 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:43.014+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:43.214 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:43.210+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:43.423 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:43.423+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:44.011 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:44.010+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:44.215 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:44.214+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:44.464 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:44.463+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:45.013 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:45.012+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:45.199 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:45.199+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:45.482 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:45.482+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:45.986 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:45.985+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:46.187 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:46.186+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:46.454 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:46.453+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:46.952 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:46.951+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:47.218 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:47.217+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:47.421 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:47.420+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:47.977 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:47.976+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:48.182 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:48.181+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:48.412 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:48.412+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:48.948 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:48.947+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:49.157 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:49.156+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:49.404 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:49.403+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:49.990 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:49.990+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:50.146 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:50.145+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:50.391 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:50.390+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:50.958 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:50.958+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:51.150 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:51.149+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:51.370 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:51.370+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:51.939 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:51.938+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:52.134 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:52.133+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:52.393 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:52.392+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:52.927 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:52.926+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:53.174 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:53.173+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:53.394 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:53.393+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:53.965 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:53.965+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:54.219 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:54.218+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:54.408 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:54.407+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:54.953 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:54.953+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:55.243 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:55.243+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:55.362 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:55.361+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:55.961 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:55.960+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:56.292 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:56.291+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:56.339 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:56.339+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:56.999 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:56.998+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:57.336 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:57.336+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:57.361 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:57.360+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:58.007 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:58.007+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:58.296 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:58.296+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:58.356 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:58.355+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:59.052 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:24:59.051+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:59.327 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:24:59.326+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:24:59.332 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:24:59.331+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:00.026 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:00.026+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:00.333 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:00.333+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:00.340 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:00.341+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:01.004 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:01.005+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:01.324 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:01.325+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:01.360 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:01.360+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:02.033 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:02.033+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:02.319 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:02.319+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:02.384 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:02.384+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:02.989 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:02.989+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:03.323 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:03.323+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:03.348 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:03.348+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:04.024 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:04.024+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:04.278 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:04.278+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:04.361 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:04.361+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:04.983 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:04.983+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:05.255 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:05.255+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:05.330 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:05.331+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:05.988 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:05.988+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:06.289 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:06.289+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:06.358 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:06.358+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:07.008 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:07.008+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:07.258 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:07.259+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:07.322 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:07.322+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:08.029 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:08.029+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:08.268 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:08.268+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:08.304 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:08.304+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:09.058 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:09.058+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:09.246 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:09.246+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:09.346 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:09.346+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:10.014 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:10.014+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:10.233 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:10.234+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:10.307 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:10.306+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:11.054 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:11.054+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:11.267 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:11.267+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:11.270 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:11.271+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:12.059 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:12.059+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:12.264 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:12.265+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:12.313 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:12.313+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:13.091 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:13.090+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:13.273 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:13.272+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:13.351 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:13.350+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:14.130 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:14.130+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:14.256 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:14.255+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:14.302 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:14.302+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:15.132 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:15.132+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:15.287 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:15.286+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:15.301 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:15.300+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:16.123 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:16.122+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:16.299 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:16.299+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:16.348 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:16.347+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:17.118 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:17.118+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:17.296 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:17.295+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:17.386 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:17.385+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:18.119 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:18.119+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:18.321 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:18.320+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:18.343 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:18.343+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:19.105 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:19.104+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:19.314 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:19.313+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:19.331 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:19.331+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:20.112 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:20.112+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:20.338 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:20.338+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:20.376 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:20.375+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:21.154 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:21.153+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:21.304 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:21.303+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:21.329 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:21.328+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:22.174 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:22.173+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:22.285 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:22.284+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:22.351 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:22.350+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:23.217 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:23.216+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:23.311 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:23.310+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:23.316 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:23.315+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:24.219 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:24.218+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:24.317 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:24.316+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:24.352 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:24.351+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:25.206 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:25.205+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:25.321 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:25.321+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:25.363 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:25.362+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:26.191 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:26.190+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:26.310 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:26.309+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:26.313 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:26.313+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:27.206 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:27.205+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:27.311 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:27.310+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:27.347 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:27.347+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:28.204 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:28.204+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:28.347 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:28.346+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:28.368 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:28.367+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:29.171 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:29.170+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:29.318 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:29.318+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:29.396 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:29.395+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:30.176 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:30.176+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:30.318 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:30.318+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:30.408 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:30.407+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:31.150 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:31.149+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:31.329 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:31.328+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:31.415 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:31.414+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:32.116 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:32.115+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:32.374 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:32.374+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:32.420 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:32.419+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:33.147 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:33.146+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:33.356 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:33.356+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:33.420 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:33.419+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:34.174 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:34.173+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:34.385 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:34.384+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:34.420 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:34.420+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:35.160 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:35.159+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:35.415 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:35.415+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:35.462 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:35.461+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:36.117 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:36.116+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:36.417 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:36.416+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:36.432 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:36.431+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:37.074 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:37.073+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:37.459 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:37.458+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:37.463 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:37.462+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:38.081 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:38.081+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:38.454 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:38.453+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:38.454 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:38.454+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:39.056 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:39.055+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:39.451 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:39.450+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:39.469 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:39.468+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:40.070 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:40.070+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:40.448 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:40.447+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:40.478 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:40.477+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:41.084 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:41.084+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:41.408 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:41.407+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:41.438 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:41.438+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:42.111 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:42.110+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:42.426 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:42.426+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:42.444 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:42.443+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:43.075 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:43.074+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:43.436 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:43.435+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:43.450 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:43.449+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:44.050 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:44.050+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:44.419 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:44.419+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:44.422 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:44.422+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:45.027 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:45.027+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:45.386 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:45.387+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:45.420 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:45.421+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:46.074 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:46.074+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:46.395 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:46.395+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:46.443 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:46.443+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:47.118 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:47.119+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:47.353 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:47.353+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:47.433 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:47.433+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:48.161 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:48.160+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:48.375 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:48.375+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:48.391 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:48.391+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:49.177 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:49.178+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:49.417 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:49.417+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:49.417 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:49.418+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:50.175 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:50.175+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:50.369 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:50.370+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:50.410 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:50.410+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:51.218 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:51.218+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:51.357 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:51.358+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:51.447 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:51.447+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:52.211 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:52.211+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:52.384 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:52.385+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:52.421 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:52.421+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:53.175 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:53.176+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:53.381 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:53.382+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:53.393 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:53.394+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:54.156 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:54.156+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:54.399 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:54.400+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:54.408 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:54.409+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:55.176 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:55.176+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:55.370 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:55.371+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:55.447 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:55.447+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:56.187 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:56.187+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:56.361 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:56.360+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:56.432 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:56.433+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:57.172 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:57.172+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:57.380 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:57.381+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:57.431 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:57.431+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:58.176 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:58.175+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:58.410 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:58.410+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:58.428 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:58.428+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:59.224 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:25:59.223+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:59.430 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:25:59.430+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:25:59.452 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:25:59.452+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:00.180 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:00.180+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:00.416 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:00.416+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:00.439 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:00.439+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:01.160 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:01.159+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:01.433 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:01.432+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:01.437 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:01.436+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:02.162 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:02.162+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:02.417 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:02.416+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:02.429 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:02.428+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:03.116 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:03.115+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:03.380 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:03.380+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:03.456 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:03.456+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:04.149 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:04.148+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:04.377 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:04.377+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:04.446 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:04.446+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:05.105 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:05.105+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:05.386 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:05.386+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:05.462 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:05.461+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:06.131 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:06.130+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:06.373 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:06.373+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:06.427 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:06.427+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:07.117 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:07.117+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:07.391 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:07.390+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:07.470 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:07.469+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:08.152 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:08.151+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:08.411 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:08.410+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:08.466 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:08.466+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:09.115 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:09.114+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:09.434 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:09.434+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:09.451 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:09.451+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:10.099 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:10.098+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:10.406 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:10.404+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:10.463 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:10.463+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:11.110 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:11.110+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:11.384 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:11.383+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:11.446 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:11.446+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:12.105 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:12.105+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:12.352 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:12.352+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:12.463 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:12.462+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:13.126 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:13.125+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:13.366 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:13.366+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:13.439 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:13.439+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:14.141 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:14.141+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:14.323 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:14.323+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:14.418 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:14.418+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:15.097 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:15.097+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:15.370 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:15.370+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:15.408 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:15.407+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:16.118 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:16.118+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:16.348 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:16.348+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:16.436 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:16.436+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:17.089 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:17.089+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:17.326 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:17.326+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:17.396 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:17.396+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:18.095 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:18.094+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:18.373 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:18.373+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:18.394 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:18.394+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:19.096 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:19.096+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:19.366 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:19.366+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:19.409 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:19.409+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:20.129 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:20.129+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:20.327 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:20.326+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:20.399 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:20.399+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:21.097 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:21.095+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:21.289 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:21.289+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:21.375 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:21.375+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:22.138 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:22.138+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:22.279 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:22.279+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:22.417 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:22.417+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:23.148 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:23.148+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:23.244 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:23.244+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:23.418 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:23.418+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:24.118 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:24.118+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:24.247 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:24.247+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:24.452 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:24.452+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:25.106 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:25.106+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:25.205 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:25.205+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:25.406 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:25.406+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:26.062 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:26.062+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:26.231 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:26.231+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:26.428 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:26.428+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:27.032 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:27.032+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:27.193 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:27.193+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:27.383 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:27.383+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:27.997 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:27.997+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:28.176 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:28.176+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:28.370 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:28.370+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:28.982 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:28.982+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:29.218 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:29.218+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:29.344 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:29.345+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:30.005 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:30.006+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:30.175 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:30.177+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:30.334 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:30.335+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:31.044 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:31.046+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:31.147 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:31.149+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:31.346 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:31.347+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:32.085 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:32.085+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:32.137 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:32.138+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:32.299 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:32.300+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:33.133 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:33.133+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:33.166 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:33.168+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:33.268 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:33.269+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:34.099 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:34.101+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:34.177 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:34.178+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:34.242 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:34.243+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:35.131 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:35.132+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:35.138 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:35.139+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:35.217 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:35.219+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:36.151 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:36.152+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:36.162 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:36.163+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:36.232 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:36.233+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:37.172 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:37.174+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:37.186 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:37.188+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:37.239 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:37.240+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:38.165 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:38.167+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:38.215 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:38.217+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:38.220 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:38.221+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:39.137 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:39.139+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:39.208 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:39.209+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:39.230 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:39.231+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:40.113 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:40.114+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:40.197 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:40.199+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:40.245 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:40.246+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:41.074 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:41.075+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:41.148 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:41.150+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:41.233 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:41.235+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:42.031 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:42.032+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:42.131 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:42.132+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:42.206 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:42.208+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:43.060 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:43.061+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:43.131 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:43.132+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:43.255 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:43.256+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:44.042 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:44.042+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:44.149 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:44.149+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:44.301 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:44.302+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:45.082 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:45.082+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:45.181 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:45.182+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:45.301 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:45.301+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:46.082 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:46.083+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:46.158 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:46.159+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:46.316 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:46.316+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:47.065 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:47.064+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:47.191 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:47.191+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:47.271 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:47.270+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:48.061 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:48.058+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:48.149 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:48.149+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:48.301 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:48.301+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:49.057 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:49.058+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:49.100 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:49.100+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:49.280 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:49.280+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:50.100 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:50.101+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:50.112 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:50.112+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:50.253 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:50.253+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:51.125 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:51.125+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:51.159 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:51.160+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:51.232 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:51.233+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:52.085 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:52.086+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:52.129 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:52.130+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:52.239 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:52.240+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:53.114 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:53.114+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:53.171 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:53.172+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:53.228 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:53.229+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:54.088 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:54.089+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:54.151 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:54.152+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:54.253 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:54.253+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:55.043 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:55.044+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:55.123 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:55.124+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:55.238 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:55.236+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:56.007 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:56.008+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:56.089 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:56.089+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:56.269 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:56.270+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:56.973 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:56.974+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:57.093 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:57.094+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:57.295 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:57.296+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:58.018 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:58.019+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:58.135 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:58.136+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:58.331 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:58.332+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:59.027 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:59.028+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:59.174 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:26:59.175+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:59.301 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:26:59.302+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:26:59.988 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:26:59.988+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:00.173 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:00.174+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:00.325 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:00.326+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:01.008 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:01.009+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:01.151 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:01.152+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:01.325 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:01.326+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:01.961 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:01.962+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:02.111 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:02.112+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:02.289 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:02.290+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:02.943 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:02.944+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:03.143 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:03.144+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:03.267 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:03.268+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:03.931 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:03.932+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:04.102 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:04.103+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:04.311 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:04.312+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:04.912 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:04.913+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:05.090 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:05.090+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:05.277 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:05.278+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:05.936 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:05.937+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:06.134 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:06.135+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:06.236 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:06.237+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:06.903 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:06.904+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:07.114 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:07.115+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:07.264 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:07.265+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:07.861 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:07.862+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:08.116 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:08.117+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:08.290 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:08.291+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:08.832 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:08.833+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:09.123 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:09.125+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:09.259 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:09.260+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:09.828 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:09.829+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:10.109 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:10.110+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:10.281 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:10.282+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:10.840 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:10.841+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:11.149 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:11.150+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:11.327 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:11.328+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:11.830 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:11.831+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:12.151 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:12.152+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:12.299 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:12.300+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:12.878 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:12.880+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:13.119 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:13.121+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:13.291 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:13.292+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:13.861 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:13.862+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:14.071 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:14.073+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:14.320 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:14.322+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:14.820 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:14.822+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:15.036 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:15.038+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:15.308 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:15.311+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:15.793 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:15.795+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:16.084 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:16.086+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:16.337 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:16.339+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:16.807 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:16.809+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:17.072 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:17.074+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:17.351 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:17.353+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:17.817 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:17.820+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:18.068 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:18.071+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:18.318 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:18.321+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:18.836 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:18.838+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:19.067 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:19.070+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:19.315 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:19.317+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:19.803 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:19.805+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:20.103 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:20.105+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:20.305 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:20.307+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:20.780 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:20.782+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:21.055 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:21.058+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:21.344 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:21.346+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:21.777 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:21.780+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:22.102 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:22.104+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:22.393 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:22.395+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:22.808 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:22.811+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:23.077 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:23.080+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:23.384 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:23.386+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:23.762 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:23.764+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:24.073 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:24.075+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:24.391 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:24.394+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:24.742 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:24.745+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:25.093 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:25.095+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:25.370 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:25.372+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:25.698 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:25.700+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:26.090 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:26.092+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:26.368 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:26.371+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:26.703 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:26.705+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:27.095 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:27.097+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:27.322 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:27.324+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:27.694 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:27.695+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:28.057 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:28.058+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:28.308 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:28.310+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:28.718 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:28.719+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:29.044 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:29.046+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:29.350 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:29.352+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:29.681 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:29.683+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:30.090 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:30.092+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:30.342 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:30.343+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:30.706 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:30.707+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:31.107 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:31.109+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:31.363 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:31.365+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:31.690 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:31.691+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:32.114 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:32.115+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:32.389 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:32.391+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:32.687 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:32.689+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:33.117 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:33.118+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:33.406 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:33.408+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:33.720 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:33.721+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:34.093 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:34.093+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:34.443 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:34.445+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:34.767 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:34.769+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:35.045 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:35.046+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:35.401 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:35.402+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:35.805 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:35.807+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:36.063 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:36.064+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:36.433 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:36.433+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:36.801 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:36.803+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:37.015 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:37.017+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:37.421 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:37.422+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:37.793 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:37.795+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:38.018 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:38.020+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:38.450 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:38.452+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:38.821 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:38.823+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:39.007 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:39.008+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:39.448 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:39.450+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:39.819 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:39.821+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:40.032 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:40.034+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:40.455 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:40.456+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:40.863 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:40.865+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:41.073 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:41.075+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:41.408 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:41.409+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:41.847 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:41.848+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:42.111 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:42.113+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:42.425 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:42.427+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:42.849 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:42.850+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:43.084 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:43.086+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:43.434 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:43.436+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:43.810 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:43.812+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:44.118 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:44.120+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:44.415 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:44.417+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:44.771 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:44.773+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:45.160 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:45.162+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:45.416 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:45.418+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:45.761 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:45.763+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:46.173 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:46.174+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:46.416 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:46.418+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:46.737 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:46.739+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:47.169 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:47.171+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:47.390 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:47.392+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:47.744 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:47.746+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:48.140 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:48.142+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:48.426 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:48.428+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:48.698 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:48.700+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:49.174 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:49.175+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:49.409 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:49.411+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:49.701 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:49.703+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:50.143 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:50.144+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:50.378 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:50.380+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:50.686 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:50.688+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:51.128 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:51.130+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:51.386 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:51.388+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:51.692 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:51.694+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:52.115 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:52.117+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:52.402 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:52.404+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:52.737 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:52.739+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:53.105 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:53.107+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:53.395 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:53.397+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:53.758 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:53.760+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:54.102 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:54.104+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:54.433 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:54.435+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:54.718 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:54.720+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:55.131 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:55.133+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:55.444 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:55.446+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:55.768 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:55.770+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:56.123 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:56.125+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:56.486 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:56.488+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:56.778 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:56.781+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:57.092 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:57.094+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:57.462 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:57.464+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:57.786 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:57.788+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:58.092 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:58.094+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:58.443 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:58.445+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:58.833 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:58.837+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:59.076 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:27:59.079+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:59.415 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:27:59.418+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:27:59.839 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:27:59.843+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:00.093 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:00.096+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:00.420 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:00.423+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:00.875 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:00.879+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:01.143 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:01.146+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:01.464 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:01.467+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:01.861 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:01.864+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:02.164 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:02.167+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:02.497 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:02.500+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:02.846 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:02.849+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:03.132 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:03.135+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:03.480 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:03.483+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:03.820 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:03.824+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:04.099 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:04.102+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:04.484 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:04.488+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:04.813 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:04.816+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:05.085 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:05.087+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:05.449 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:05.452+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:05.836 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:05.839+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:06.075 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:06.078+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:06.496 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:06.500+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:06.870 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:06.873+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:07.109 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:07.112+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:07.542 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:07.542+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:07.870 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:07.873+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:08.080 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:08.084+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:08.530 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:08.534+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:08.873 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:08.876+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:09.041 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:09.045+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:09.515 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:09.518+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:09.908 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:09.911+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:10.070 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:10.074+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:10.538 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:10.542+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:10.887 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:10.891+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:11.110 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:11.113+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:11.528 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:11.532+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:11.903 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:11.907+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:12.151 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:12.154+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:12.508 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:12.511+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:12.881 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:12.884+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:13.175 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:13.177+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:13.500 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:13.503+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:13.849 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:13.851+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:14.157 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:14.159+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:14.538 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:14.541+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:14.803 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:14.806+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:15.199 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:15.202+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:15.553 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:15.555+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:15.763 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:15.765+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:16.183 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:16.186+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:16.567 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:16.570+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:16.737 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:16.740+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:17.208 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:17.210+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:17.593 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:17.595+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:17.787 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:17.790+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:18.158 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:18.161+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:18.618 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:18.621+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:18.790 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:18.792+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:19.155 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:19.157+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:19.638 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:19.640+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:19.832 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:19.834+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:20.110 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:20.113+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:20.611 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:20.614+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:20.866 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:20.869+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:21.106 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:21.108+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:21.609 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:21.611+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:21.832 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:21.834+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:22.117 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:22.120+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:22.608 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:22.610+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:22.807 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:22.810+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:23.094 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:23.096+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:23.590 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:23.593+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:23.787 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:23.789+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:24.091 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:24.094+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:24.547 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:24.549+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:24.829 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:24.831+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:25.062 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:25.065+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:25.538 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:25.541+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:25.795 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:25.797+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:26.050 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:26.053+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:26.511 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:26.514+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:26.839 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:26.841+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:27.014 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:27.017+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:27.532 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:27.534+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:27.790 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:27.793+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:28.042 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:28.045+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:28.508 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:28.511+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:28.804 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:28.807+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:29.063 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:29.066+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:29.497 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:29.500+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:29.844 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:29.846+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:30.060 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:30.063+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:30.531 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:30.534+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:30.842 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:30.844+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:31.094 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:31.097+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:31.570 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:31.573+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:31.846 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:31.849+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:32.088 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:32.091+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:32.597 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:32.600+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:32.816 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:32.819+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:33.083 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:33.086+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:33.649 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:33.649+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:33.786 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:33.789+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:34.130 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:34.133+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:34.644 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:34.647+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:34.835 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:34.838+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:35.104 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:35.107+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:35.650 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:35.653+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:35.812 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:35.815+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:36.072 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:36.075+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:36.688 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:36.691+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:36.762 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:36.765+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:37.094 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:37.097+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:37.734 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:37.737+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:37.792 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:37.795+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:38.077 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:38.079+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:38.703 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:38.706+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:38.776 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:38.779+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:39.079 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:39.082+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:39.680 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:39.683+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:39.729 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:39.732+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:40.060 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:40.063+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:40.694 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:40.697+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:40.757 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:40.760+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:41.109 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:41.112+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:41.690 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:41.692+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:41.776 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:41.779+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:42.072 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:42.075+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:42.703 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:42.706+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:42.797 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:42.800+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:43.073 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:43.077+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:43.733 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:43.736+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:43.819 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:43.822+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:44.029 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:44.032+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:44.736 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:44.740+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:44.860 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:44.864+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:45.033 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:45.037+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:45.740 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:45.743+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:45.853 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:45.857+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:46.051 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:46.056+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:46.783 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:46.787+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:46.811 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:46.815+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:47.074 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:47.078+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:47.792 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:47.796+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:47.810 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:47.814+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:48.049 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:48.053+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:48.822 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:48.826+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:48.822 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:48.826+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:49.045 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:49.049+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:49.791 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:49.795+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:49.819 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:49.824+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:50.058 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:50.062+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:50.792 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:50.797+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:50.829 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:50.834+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:51.073 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:51.078+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:51.751 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:51.755+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:51.797 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:51.802+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:52.051 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:52.056+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:52.734 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:52.738+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:52.791 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:52.795+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:53.053 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:53.058+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:53.735 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:53.739+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:53.746 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:53.750+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:54.099 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:54.104+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:54.720 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:54.725+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:54.773 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:54.777+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:55.089 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:55.094+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:55.748 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:55.752+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:55.763 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:55.768+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:56.080 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:56.085+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:56.750 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:56.754+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:56.761 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:56.765+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:57.068 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:57.072+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:57.718 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:57.722+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:57.805 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:57.783+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:58.109 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:58.113+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:58.708 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:58.712+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:58.789 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:58.793+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:59.123 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:28:59.126+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:59.745 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:28:59.749+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:28:59.820 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:28:59.824+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:00.153 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:00.157+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:00.790 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:00.794+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:00.840 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:00.843+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:01.133 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:01.137+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:01.748 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:01.751+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:01.826 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:01.829+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:02.112 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:02.115+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:02.751 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:02.755+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:02.817 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:02.821+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:03.098 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:03.101+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:03.775 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:03.779+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:03.862 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:03.866+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:04.053 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:04.057+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:04.808 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:04.811+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:04.819 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:04.823+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:05.086 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:05.090+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:05.799 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:05.803+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:05.822 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:05.822+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:06.135 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:06.139+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:06.802 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:06.806+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:06.841 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:06.845+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:07.155 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:07.159+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:07.804 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:07.807+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:07.807 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:07.810+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:08.179 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:08.183+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:08.838 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:08.841+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:08.843 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:08.847+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:09.229 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:09.233+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:09.809 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:09.813+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:09.815 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:09.819+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:10.279 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:10.283+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:10.776 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:10.779+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:10.834 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:10.838+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:11.231 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:11.235+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:11.728 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:11.731+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:11.860 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:11.863+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:12.192 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:12.196+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:12.743 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:12.747+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:12.841 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:12.845+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:13.157 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:13.161+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:13.780 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:13.784+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:13.824 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:13.828+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:14.156 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:14.159+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:14.744 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:14.748+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:14.842 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:14.846+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:15.119 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:15.123+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:15.793 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:15.796+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:15.874 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:15.878+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:16.093 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:16.097+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:16.776 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:16.780+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:16.899 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:16.903+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:17.098 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:17.102+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:17.750 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:17.749+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:17.874 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:17.878+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:18.091 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:18.095+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:18.775 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:18.775+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:18.836 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:18.839+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:19.093 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:19.097+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:19.790 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:19.794+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:19.814 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:19.817+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:20.085 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:20.089+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:20.752 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:20.756+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:20.835 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:20.839+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:21.107 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:21.111+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:21.752 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:21.755+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:21.866 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:21.870+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:22.062 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:22.066+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:22.735 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:22.739+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:22.849 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:22.853+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:23.088 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:23.091+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:23.765 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:23.769+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:23.809 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:23.813+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:24.082 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:24.086+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:24.761 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:24.765+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:24.791 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:24.795+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:25.038 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:25.042+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:25.797 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:25.801+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:25.834 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:25.838+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:26.045 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:26.049+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:26.794 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:26.798+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:26.873 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:26.877+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:27.084 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:27.088+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:27.839 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:27.843+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:27.892 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:27.896+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:28.074 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:28.077+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:28.870 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:28.874+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:28.900 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:28.905+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:29.087 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:29.092+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:29.851 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:29.856+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:29.940 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:29.945+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:30.129 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:30.134+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:30.805 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:30.810+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:30.910 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:30.915+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:31.126 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:31.131+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:31.781 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:31.786+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:31.933 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:31.939+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:32.173 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:32.178+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:32.756 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:32.762+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:32.958 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:32.963+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:33.211 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:33.216+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:33.770 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:33.776+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:33.944 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:33.949+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:34.188 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:34.193+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:34.788 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:34.793+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:34.958 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:34.963+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:35.163 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:35.168+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:35.741 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:35.746+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:36.000 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:36.005+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:36.118 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:36.123+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:36.783 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:36.788+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:37.007 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:37.012+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:37.107 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:37.112+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:37.792 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:37.797+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:37.972 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:37.978+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:38.140 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:38.146+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:38.791 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:38.796+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:38.944 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:38.949+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:39.135 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:39.140+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:39.817 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:39.822+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:39.975 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:39.981+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:40.170 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:40.175+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:40.828 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:40.832+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:40.996 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:41.002+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:41.154 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:41.160+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:41.804 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:41.809+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:41.974 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:41.979+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:42.110 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:42.114+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:42.844 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:42.849+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:43.014 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:43.019+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:43.082 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:43.087+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:43.863 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:43.867+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:44.044 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:44.048+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:44.114 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:44.118+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:44.850 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:44.855+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:45.056 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:45.061+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:45.138 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:45.143+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:45.877 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:45.881+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:46.040 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:46.044+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:46.114 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:46.119+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:46.874 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:46.878+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:47.080 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:47.085+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:47.084 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:47.089+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:47.921 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:47.925+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:48.078 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:48.082+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:48.118 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:48.123+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:48.889 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:48.893+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:49.076 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:49.080+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:49.084 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:49.089+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:49.936 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:49.941+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:50.044 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:50.048+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:50.103 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:50.108+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:50.986 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:50.990+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:51.083 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:51.087+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:51.088 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:51.093+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:51.982 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:51.987+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:52.072 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:52.076+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:52.129 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:52.133+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:52.982 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:52.987+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:53.078 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:53.083+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:53.142 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:53.147+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:53.995 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:53.999+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:54.113 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:54.117+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:54.182 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:54.187+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:55.005 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:55.009+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:55.119 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:55.124+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:55.150 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:55.155+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:55.958 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:55.963+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:56.133 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:56.137+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:56.184 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:56.189+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:56.968 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:56.972+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:57.170 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:57.173+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:57.215 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:57.219+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:57.926 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:57.930+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:58.189 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:58.194+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:58.195 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:58.200+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:58.958 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:58.962+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:59.192 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:29:59.197+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:59.224 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:29:59.228+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:29:59.974 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:29:59.979+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:00.192 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:00.196+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:00.217 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:00.222+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:00.969 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:00.973+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:01.152 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:01.157+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:01.221 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:01.226+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:01.920 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:01.925+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:02.121 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:02.126+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:02.181 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:02.183+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:02.949 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:02.953+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:03.126 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:03.131+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:03.224 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:03.229+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:03.961 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:03.966+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:04.091 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:04.096+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:04.230 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:04.234+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:04.924 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:04.929+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:05.113 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:05.117+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:05.208 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:05.212+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:05.950 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:05.955+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:06.128 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:06.133+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:06.174 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:06.179+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:06.982 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:06.986+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:07.158 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:07.163+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:07.160 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:07.165+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:08.029 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:08.033+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:08.144 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:08.149+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:08.179 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:08.183+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:09.051 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:09.055+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:09.133 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:09.137+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:09.179 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:09.183+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:10.015 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:10.019+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:10.098 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:10.103+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:10.193 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:10.198+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:10.986 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:10.991+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:11.103 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:11.108+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:11.233 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:11.238+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:11.941 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:11.947+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:12.069 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:12.074+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:12.235 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:12.240+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:12.909 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:12.914+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:13.119 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:13.124+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:13.199 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:13.203+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:13.896 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:13.902+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:14.158 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:14.163+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:14.233 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:14.239+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:14.891 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:14.896+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:15.162 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:15.167+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:15.249 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:15.255+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:15.886 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:15.892+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:16.160 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:16.166+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:16.297 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:16.302+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:16.858 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:16.864+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:17.197 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:17.203+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:17.297 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:17.303+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:17.831 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:17.837+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:18.183 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:18.189+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:18.255 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:18.261+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:18.872 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:18.878+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:19.154 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:19.160+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:19.249 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:19.255+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:19.883 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:19.889+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:20.151 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:20.157+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:20.206 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:20.212+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:20.861 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:20.867+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:21.123 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:21.129+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:21.251 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:21.257+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:21.891 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:21.897+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:22.147 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:22.153+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:22.285 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:22.290+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:22.897 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:22.900+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:23.101 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:23.106+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:23.276 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:23.281+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:23.846 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:23.852+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:24.089 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:24.095+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:24.238 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:24.244+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:24.833 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:24.839+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:25.053 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:25.059+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:25.280 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:25.286+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:25.801 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:25.807+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:26.037 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:26.043+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:26.302 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:26.308+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:26.825 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:26.831+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:27.011 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:27.016+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:27.313 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:27.318+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:27.811 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:27.816+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:28.006 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:28.010+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:28.363 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:28.368+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:28.777 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:28.782+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:29.039 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:29.044+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:29.329 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:29.334+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:29.822 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:29.827+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:29.995 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:30.000+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:30.289 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:30.294+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:30.784 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:30.789+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:30.975 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:30.980+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:31.248 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:31.254+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:31.755 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:31.760+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:31.952 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:31.957+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:32.224 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:32.229+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:32.729 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:32.734+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:32.921 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:32.926+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:33.214 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:33.219+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:33.695 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:33.700+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:33.895 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:33.900+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:34.198 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:34.203+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:34.678 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:34.683+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:34.903 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:34.908+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:35.162 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:35.168+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:35.725 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:35.730+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:35.855 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:35.859+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:36.117 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:36.122+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:36.748 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:36.754+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:36.821 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:36.826+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:37.092 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:37.097+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:37.787 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:37.792+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:37.844 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:37.849+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:38.074 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:38.079+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:38.752 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:38.757+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:38.855 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:38.860+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:39.110 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:39.115+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:39.800 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:39.805+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:39.843 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:39.848+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:40.128 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:40.134+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:40.831 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:40.836+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:40.833 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:40.838+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:41.170 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:41.173+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:41.832 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:41.837+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:41.850 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:41.856+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:42.184 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:42.189+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:42.827 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:42.832+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:42.858 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:42.864+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:43.213 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:43.218+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:43.832 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:43.838+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:43.876 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:43.882+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:44.180 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:44.185+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:44.816 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:44.821+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:44.908 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:44.914+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:45.144 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:45.149+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:45.807 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:45.812+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:45.918 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:45.923+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:46.181 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:46.186+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:46.841 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:46.846+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:46.944 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:46.950+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:47.186 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:47.191+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:47.838 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:47.843+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:47.912 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:47.918+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:48.171 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:48.176+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:48.873 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:48.878+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:48.886 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:48.891+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:49.187 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:49.192+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:49.874 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:49.880+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:49.909 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:49.915+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:50.166 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:50.171+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:50.905 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:50.911+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:50.929 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:50.934+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:51.194 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:51.199+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:51.899 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:51.904+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:51.929 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:51.934+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:52.152 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:52.157+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:52.922 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:52.927+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:52.923 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:52.929+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:53.149 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:53.155+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:53.950 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:53.955+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:53.960 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:53.966+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:54.118 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:54.123+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:54.991 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:54.996+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:55.004 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:55.009+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:55.108 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:55.114+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:55.980 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:55.985+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:56.003 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:56.009+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:56.078 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:56.084+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:56.975 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:56.980+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:57.012 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:57.018+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:57.067 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:57.073+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:57.941 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:57.947+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:57.997 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:58.003+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:58.074 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:58.081+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:58.936 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:58.942+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:58.963 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:58.970+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:59.026 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:30:59.033+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:59.949 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:30:59.956+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:30:59.974 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:30:59.981+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:00.028 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:00.035+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:00.999 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:01.006+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:01.016 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:01.022+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:01.018 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:01.025+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:01.977 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:01.983+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:02.001 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:02.008+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:02.011 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:02.018+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:02.938 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:02.945+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:02.963 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:02.969+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:03.033 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:03.039+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:03.960 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:03.967+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:03.964 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:03.971+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:04.049 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:04.056+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:04.964 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:04.970+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:05.001 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:05.008+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:05.053 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:05.060+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:05.967 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:05.974+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:06.005 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:06.012+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:06.038 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:06.044+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:06.947 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:06.954+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:07.020 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:07.026+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:07.087 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:07.093+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:07.928 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:07.934+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:08.011 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:08.018+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:08.080 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:08.087+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:08.926 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:08.933+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:08.980 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:08.986+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:09.067 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:09.073+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:09.964 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:09.971+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:09.979 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:09.985+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:10.083 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:10.090+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:10.935 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:10.942+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:11.023 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:11.030+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:11.119 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:11.126+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:11.931 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:11.936+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:12.042 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:12.048+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:12.156 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:12.162+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:12.951 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:12.956+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:13.084 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:13.089+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:13.199 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:13.205+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:13.916 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:13.922+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:14.078 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:14.083+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:14.215 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:14.221+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:14.946 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:14.951+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:15.063 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:15.069+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:15.169 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:15.175+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:15.906 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:15.912+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:16.066 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:16.072+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:16.147 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:16.153+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:16.889 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:16.895+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:17.036 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:17.042+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:17.155 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:17.161+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:17.874 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:17.879+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:18.083 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:18.088+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:18.150 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:18.156+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:18.921 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:18.926+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:19.113 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:19.119+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:19.141 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:19.147+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:19.891 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:19.897+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:20.067 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:20.073+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:20.116 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:20.122+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:20.921 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:20.927+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:21.037 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:21.043+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:21.148 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:21.154+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:21.971 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:21.977+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:22.004 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:22.010+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:22.113 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:22.119+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:22.926 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:22.931+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:23.026 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:23.032+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:23.090 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:23.096+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:23.917 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:23.923+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:23.982 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:23.988+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:24.105 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:24.111+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:24.899 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:24.905+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:24.984 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:24.990+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:25.101 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:25.107+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:25.861 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:25.866+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:25.984 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:25.990+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:26.140 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:26.145+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:26.872 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:26.878+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:27.001 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:27.007+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:27.177 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:27.183+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:27.895 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:27.901+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:27.952 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:27.958+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:28.218 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:28.223+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:28.861 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:28.867+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:29.002 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:29.008+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:29.214 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:29.220+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:29.836 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:29.842+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:29.973 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:29.979+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:30.195 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:30.201+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:30.881 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:30.886+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:30.945 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:30.951+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:31.183 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:31.189+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:31.929 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:31.935+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:31.982 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:31.989+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:32.173 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:32.179+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:32.944 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:32.950+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:32.990 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:32.996+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:33.213 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:33.219+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:33.980 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:33.986+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:34.029 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:34.035+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:34.241 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:34.247+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:35.007 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:35.013+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:35.015 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:35.022+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:35.208 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:35.214+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:35.984 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:35.990+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:35.990 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:35.997+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:36.223 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:36.229+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:36.943 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:36.949+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:36.946 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:36.953+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:37.206 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:37.212+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:37.906 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:37.913+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:37.993 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:37.999+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:38.243 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:38.249+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:38.906 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:38.912+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:38.965 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:38.971+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:39.199 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:39.205+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:39.928 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:39.934+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:40.008 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:40.014+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:40.204 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:40.210+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:40.961 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:40.967+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:40.996 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:41.002+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:41.204 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:41.211+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:41.981 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:41.988+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:42.010 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:42.016+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:42.176 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:42.183+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:42.975 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:42.981+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:43.007 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:43.013+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:43.211 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:43.218+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:44.001 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:44.008+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:44.021 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:44.029+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:44.192 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:44.200+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:44.951 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:44.959+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:44.985 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:44.993+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:45.188 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:45.196+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:45.943 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:45.951+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:45.959 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:45.967+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:46.186 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:46.193+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:46.915 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:46.922+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:46.983 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:46.990+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:47.236 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:47.243+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:47.899 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:47.906+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:47.946 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:47.954+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:48.193 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:48.200+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:48.883 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:48.889+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:48.958 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:48.965+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:49.165 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:49.172+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:49.893 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:49.900+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:49.950 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:49.957+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:50.196 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:50.203+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:50.941 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:50.949+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:50.962 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:50.969+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:51.244 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:51.252+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:51.898 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:51.905+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:51.961 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:51.969+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:52.230 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:52.238+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:52.916 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:52.923+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:52.917 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:52.924+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:53.205 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:53.212+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:53.904 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:53.911+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:53.940 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:53.947+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:54.179 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:54.187+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:54.892 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:54.900+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:54.953 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:54.961+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:55.210 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:55.217+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:55.942 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:55.950+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:55.970 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:55.977+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:56.229 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:56.236+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:56.965 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:56.971+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:57.006 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:57.012+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:57.264 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:57.270+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:57.965 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:57.972+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:58.002 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:58.008+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:58.234 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:58.241+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:58.941 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:58.948+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:59.032 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:31:59.038+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:59.202 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:31:59.207+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:31:59.955 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:31:59.961+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:00.017 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:00.024+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:00.173 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:00.178+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:00.911 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:00.918+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:01.050 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:01.057+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:01.149 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:01.156+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:01.940 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:01.946+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:02.047 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:02.054+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:02.155 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:02.162+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:02.990 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:02.995+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:03.008 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:03.014+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:03.145 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:03.152+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:04.001 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:04.007+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:04.047 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:04.053+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:04.161 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:04.167+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:05.048 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:05.055+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:05.076 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:05.082+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:05.162 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:05.169+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:06.033 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:06.040+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:06.080 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:06.087+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:06.115 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:06.122+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:07.065 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:07.071+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:07.073 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:07.079+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:07.161 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:07.165+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:08.059 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:08.063+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:08.109 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:08.115+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:08.128 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:08.134+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:09.027 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:09.034+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:09.132 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:09.139+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:09.135 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:09.142+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:10.065 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:10.072+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:10.093 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:10.100+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:10.163 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:10.170+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:11.069 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:11.076+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:11.097 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:11.103+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:11.136 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:11.143+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:12.027 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:12.033+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:12.085 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:12.092+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:12.166 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:12.173+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:13.001 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:13.006+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:13.114 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:13.121+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:13.212 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:13.218+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:14.020 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:14.027+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:14.067 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:14.074+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:14.176 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:14.183+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:14.994 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:15.001+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:15.027 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:15.034+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:15.173 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:15.180+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:15.985 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:15.991+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:16.016 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:16.023+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:16.193 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:16.200+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:16.991 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:16.998+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:17.027 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:17.034+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:17.199 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:17.205+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:17.957 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:17.964+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:18.015 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:18.022+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:18.191 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:18.198+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:18.918 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:18.924+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:19.045 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:19.052+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:19.223 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:19.230+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:19.924 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:19.930+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:20.047 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:20.054+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:20.192 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:20.199+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:20.922 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:20.928+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:21.088 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:21.094+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:21.199 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:21.206+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:21.909 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:21.916+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:22.091 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:22.098+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:22.218 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:22.225+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:22.891 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:22.898+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:23.137 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:23.143+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:23.214 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:23.221+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:23.918 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:23.925+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:24.134 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:24.141+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:24.184 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:24.191+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:24.936 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:24.943+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:25.105 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:25.112+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:25.215 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:25.222+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:25.898 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:25.905+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:26.063 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:26.070+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:26.189 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:26.195+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:26.900 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:26.908+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:27.097 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:27.104+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:27.143 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:27.150+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:27.918 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:27.925+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:28.078 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:28.086+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:28.160 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:28.167+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:28.911 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:28.919+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:29.097 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:29.104+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:29.156 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:29.164+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:29.888 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:29.896+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:30.081 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:30.089+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:30.117 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:30.125+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:30.851 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:30.859+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:31.107 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:31.115+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:31.110 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:31.118+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:31.895 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:31.903+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:32.107 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:32.115+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:32.116 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:32.123+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:32.864 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:32.872+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:33.077 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:33.085+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:33.158 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:33.166+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:33.858 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:33.866+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:34.054 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:34.062+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:34.127 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:34.135+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:34.826 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:34.834+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:35.015 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:35.023+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:35.158 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:35.165+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:35.834 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:35.842+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:36.053 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:36.061+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:36.208 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:36.216+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:36.878 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:36.886+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:37.079 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:37.087+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:37.246 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:37.254+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:37.888 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:37.896+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:38.077 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:38.085+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:38.254 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:38.262+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:38.881 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:38.889+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:39.100 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:39.108+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:39.210 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:39.218+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:39.889 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:39.897+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:40.068 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:40.076+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:40.190 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:40.197+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:40.934 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:40.942+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:41.079 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:41.087+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:41.231 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:41.239+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:41.950 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:41.957+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:42.129 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:42.136+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:42.278 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:42.285+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:42.970 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:42.977+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:43.153 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:43.160+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:43.269 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:43.276+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:44.016 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:44.023+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:44.124 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:44.131+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:44.291 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:44.298+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:45.059 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:45.066+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:45.158 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:45.165+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:45.337 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:45.344+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:46.074 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:46.081+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:46.133 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:46.140+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:46.331 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:46.338+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:47.062 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:47.070+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:47.087 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:47.095+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:47.296 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:47.303+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:48.055 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:48.062+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:48.064 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:48.072+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:48.346 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:48.354+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:49.067 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:49.074+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:49.073 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:49.080+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:49.348 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:49.356+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:50.081 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:50.088+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:50.089 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:50.097+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:50.388 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:50.396+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:51.122 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:51.129+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:51.126 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:51.133+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:51.345 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:51.352+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:52.119 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:52.126+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:52.160 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:52.167+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:52.337 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:52.344+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:53.096 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:53.103+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:53.208 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:53.216+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:53.351 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:53.358+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:54.093 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:54.100+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:54.257 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:54.265+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:54.391 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:54.398+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:55.117 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:55.125+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:55.211 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:55.219+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:55.349 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:55.356+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:56.163 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:56.170+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:56.163 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:56.171+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:56.373 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:56.380+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:57.133 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:57.141+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:57.212 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:57.218+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:57.359 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:57.366+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:58.112 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:58.119+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:58.200 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:58.207+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:58.310 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:58.317+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:59.157 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:32:59.165+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:59.205 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:32:59.212+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:32:59.323 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:32:59.331+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:00.118 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:00.125+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:00.165 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:00.173+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:00.356 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:00.363+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:01.072 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:01.079+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:01.210 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:01.217+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:01.319 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:01.326+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:02.051 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:02.059+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:02.188 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:02.195+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:02.347 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:02.354+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:03.005 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:03.012+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:03.187 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:03.195+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:03.370 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:03.378+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:03.959 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:03.966+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:04.160 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:04.168+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:04.339 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:04.347+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:04.916 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:04.924+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:05.138 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:05.145+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:05.310 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:05.317+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:05.957 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:05.964+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:06.141 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:06.149+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:06.274 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:06.282+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:06.968 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:06.976+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:07.133 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:07.140+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:07.315 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:07.322+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:07.998 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:08.005+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:08.087 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:08.095+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:08.296 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:08.304+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:08.985 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:08.993+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:09.066 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:09.073+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:09.290 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:09.297+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:09.962 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:09.970+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:10.033 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:10.041+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:10.292 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:10.299+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:11.001 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:11.009+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:11.035 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:11.042+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:11.341 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:11.348+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:12.020 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:12.028+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:12.082 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:12.089+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:12.351 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:12.358+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:13.067 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:13.075+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:13.086 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:13.094+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:13.322 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:13.330+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:14.115 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:14.123+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:14.126 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:14.134+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:14.311 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:14.319+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:15.103 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:15.111+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:15.160 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:15.169+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:15.319 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:15.328+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:16.055 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:16.064+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:16.196 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:16.205+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:16.360 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:16.369+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:17.047 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:17.055+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:17.214 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:17.223+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:17.388 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:17.396+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:18.086 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:18.094+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:18.203 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:18.212+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:18.388 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:18.397+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:19.124 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:19.133+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:19.240 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:19.249+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:19.367 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:19.375+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:20.127 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:20.136+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:20.289 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:20.297+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:20.331 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:20.340+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:21.105 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:21.114+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:21.264 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:21.273+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:21.331 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:21.339+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:22.098 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:22.107+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:22.305 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:22.314+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:22.312 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:22.320+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:23.130 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:23.138+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:23.312 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:23.321+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:23.331 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:23.340+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:24.149 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:24.158+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:24.276 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:24.284+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:24.308 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:24.317+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:25.105 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:25.113+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:25.295 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:25.303+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:25.323 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:25.332+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:26.113 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:26.121+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:26.285 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:26.293+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:26.304 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:26.312+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:27.105 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:27.112+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:27.268 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:27.275+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:27.349 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:27.356+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:28.080 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:28.088+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:28.258 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:28.265+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:28.299 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:28.307+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:29.085 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:29.093+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:29.238 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:29.246+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:29.336 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:29.343+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:30.063 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:30.071+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:30.248 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:30.256+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:30.335 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:30.343+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:31.044 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:31.051+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:31.216 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:31.224+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:31.289 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:31.296+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:32.069 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:32.077+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:32.249 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:32.257+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:32.311 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:32.319+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:33.111 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:33.119+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:33.240 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:33.248+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:33.290 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:33.298+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:34.092 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:34.099+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:34.218 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:34.226+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:34.289 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:34.297+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:35.063 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:35.071+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:35.254 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:35.262+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:35.283 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:35.291+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:36.074 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:36.081+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:36.239 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:36.247+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:36.293 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:36.301+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:37.058 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:37.065+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:37.277 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:37.284+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:37.284 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:37.292+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:38.033 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:38.040+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:38.275 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:38.283+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:38.317 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:38.324+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:38.995 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:39.003+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:39.229 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:39.237+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:39.300 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:39.308+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:39.956 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:39.964+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:40.267 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:40.275+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:40.320 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:40.328+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:40.965 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:40.973+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:41.277 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:41.282+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:41.298 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:41.306+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:41.939 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:41.947+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:42.239 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:42.247+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:42.269 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:42.277+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:42.947 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:42.955+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:43.230 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:43.238+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:43.283 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:43.291+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:43.901 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:43.909+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:44.239 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:44.247+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:44.267 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:44.275+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:44.861 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:44.869+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:45.247 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:45.255+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:45.297 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:45.305+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:45.823 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:45.831+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:46.240 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:46.247+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:46.326 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:46.334+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:46.855 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:46.863+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:47.227 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:47.235+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:47.331 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:47.339+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:47.857 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:47.865+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:48.265 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:48.273+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:48.357 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:48.365+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:48.894 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:48.902+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:49.268 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:49.276+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:49.323 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:49.331+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:49.936 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:49.944+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:50.228 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:50.236+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:50.350 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:50.358+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:50.916 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:50.924+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:51.255 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:51.263+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:51.383 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:51.391+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:51.901 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:51.909+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:52.261 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:52.269+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:52.431 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:52.439+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:52.859 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:52.867+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:53.269 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:53.277+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:53.393 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:53.401+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:53.830 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:53.838+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:54.290 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:54.298+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:54.439 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:54.447+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:54.816 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:54.824+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:55.323 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:55.331+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:55.465 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:55.473+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:55.767 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:55.775+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:56.277 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:56.286+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:56.492 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:56.500+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:56.750 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:56.758+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:57.246 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:57.255+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:57.481 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:57.489+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:57.706 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:57.714+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:58.222 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:58.231+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:58.436 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:58.445+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:58.724 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:58.734+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:59.251 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:33:59.260+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:59.461 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:33:59.470+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:33:59.723 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:33:59.732+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:00.301 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:00.310+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:00.459 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:00.468+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:00.755 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:00.764+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:01.289 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:01.298+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:01.488 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:01.497+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:01.757 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:01.766+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:02.248 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:02.257+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:02.471 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:02.480+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:02.745 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:02.754+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:03.213 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:03.222+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:03.433 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:03.442+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:03.749 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:03.758+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:04.168 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:04.177+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:04.388 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:04.398+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:04.718 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:04.727+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:05.167 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:05.176+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:05.348 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:05.357+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:05.738 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:05.747+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:06.129 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:06.139+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:06.357 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:06.366+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:06.769 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:06.778+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:07.088 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:07.097+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:07.341 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:07.350+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:07.777 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:07.786+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:08.040 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:08.049+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:08.354 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:08.363+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:08.776 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:08.786+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:09.056 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:09.065+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:09.310 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:09.319+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:09.801 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:09.810+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:10.079 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:10.088+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:10.352 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:10.362+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:10.834 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:10.844+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:11.105 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:11.113+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:11.319 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:11.328+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:11.830 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:11.839+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:12.124 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:12.132+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:12.297 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:12.305+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:12.834 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:12.842+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:13.081 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:13.090+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:13.303 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:13.311+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:13.814 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:13.822+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:14.130 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:14.139+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:14.279 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:14.287+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:14.810 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:14.818+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:15.090 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:15.098+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:15.298 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:15.307+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:15.795 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:15.803+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:16.078 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:16.086+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:16.346 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:16.354+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:16.802 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:16.810+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:17.101 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:17.109+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:17.335 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:17.343+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:17.762 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:17.770+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:18.138 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:18.146+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:18.340 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:18.349+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:18.791 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:18.799+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:19.164 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:19.172+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:19.306 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:19.315+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:19.795 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:19.804+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:20.145 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:20.153+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:20.281 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:20.289+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:20.767 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:20.775+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:21.133 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:21.141+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:21.330 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:21.338+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:21.807 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:21.816+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:22.178 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:22.186+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:22.282 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:22.290+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:22.786 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:22.795+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:23.210 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:23.218+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:23.271 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:23.279+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:23.779 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:23.787+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:24.232 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:24.240+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:24.303 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:24.311+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:24.821 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:24.829+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:25.260 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:25.268+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:25.290 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:25.298+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:25.780 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:25.789+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:26.228 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:26.237+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:26.264 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:26.272+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:26.766 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:26.774+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:27.239 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:27.247+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:27.277 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:27.286+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:27.764 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:27.773+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:28.202 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:28.210+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:28.323 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:28.331+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:28.794 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:28.802+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:29.247 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:29.256+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:29.345 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:29.354+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:29.820 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:29.829+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:30.209 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:30.215+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:30.373 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:30.382+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:30.806 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:30.814+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:31.250 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:31.259+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:31.345 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:31.354+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:31.797 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:31.805+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:32.236 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:32.245+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:32.307 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:32.315+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:32.813 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:32.821+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:33.201 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:33.209+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:33.285 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:33.293+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:33.805 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:33.813+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:34.190 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:34.199+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:34.282 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:34.290+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:34.811 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:34.819+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:35.217 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:35.225+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:35.323 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:35.331+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:35.784 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:35.793+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:36.172 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:36.181+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:36.349 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:36.357+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:36.750 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:36.759+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:37.186 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:37.194+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:37.356 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:37.364+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:37.714 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:37.722+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:38.144 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:38.152+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:38.354 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:38.362+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:38.741 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:38.750+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:39.177 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:39.185+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:39.385 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:39.393+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:39.714 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:39.722+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:40.129 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:40.137+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:40.344 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:40.354+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:40.693 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:40.702+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:41.092 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:41.100+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:41.330 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:41.339+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:41.680 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:41.689+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:42.060 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:42.069+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:42.332 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:42.342+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:42.692 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:42.701+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:43.109 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:43.118+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:43.292 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:43.301+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:43.738 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:43.748+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:44.136 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:44.146+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:44.288 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:44.297+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:44.690 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:44.700+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:45.140 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:45.150+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:45.249 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:45.259+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:45.732 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:45.742+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:46.151 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:46.160+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:46.285 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:46.295+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:46.763 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:46.773+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:47.152 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:47.162+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:47.293 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:47.303+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:47.770 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:47.779+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:48.173 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:48.183+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:48.260 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:48.269+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:48.772 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:48.781+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:49.156 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:49.166+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:49.212 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:49.222+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:49.819 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:49.829+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:50.138 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:50.148+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:50.202 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:50.211+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:50.843 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:50.852+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:51.184 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:51.193+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:51.231 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:51.241+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:51.859 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:51.869+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:52.211 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:52.221+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:52.265 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:52.274+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:52.890 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:52.899+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:53.163 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:53.172+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:53.298 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:53.308+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:53.854 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:53.863+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:54.205 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:54.215+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:54.279 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:54.289+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:54.808 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:54.818+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:55.237 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:55.246+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:55.275 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:55.285+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:55.826 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:55.834+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:56.215 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:56.224+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:56.227 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:56.235+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:56.812 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:56.821+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:57.173 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:57.182+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:57.214 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:57.223+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:57.836 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:57.845+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:58.211 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:58.220+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:58.224 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:58.232+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:58.845 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:58.853+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:59.211 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:34:59.220+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:59.270 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:34:59.279+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:34:59.835 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:34:59.844+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:00.211 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:00.219+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:00.300 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:00.309+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:00.877 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:00.886+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:01.213 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:01.222+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:01.302 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:01.311+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:01.836 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:01.845+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:02.174 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:02.183+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:02.305 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:02.313+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:02.833 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:02.842+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:03.223 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:03.232+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:03.329 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:03.338+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:03.805 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:03.814+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:04.220 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:04.229+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:04.350 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:04.359+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:04.837 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:04.846+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:05.224 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:05.233+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:05.387 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:05.396+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:05.880 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:05.888+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:06.203 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:06.212+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:06.372 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:06.381+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:06.890 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:06.899+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:07.168 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:07.177+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:07.350 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:07.359+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:07.898 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:07.907+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:08.127 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:08.135+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:08.380 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:08.389+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:08.869 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:08.878+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:09.126 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:09.135+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:09.354 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:09.363+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:09.878 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:09.887+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:10.174 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:10.183+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:10.374 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:10.383+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:10.919 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:10.928+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:11.194 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:11.203+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:11.375 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:11.384+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:11.957 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:11.965+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:12.174 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:12.183+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:12.360 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:12.369+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:12.999 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:13.008+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:13.198 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:13.207+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:13.335 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:13.344+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:14.028 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:14.037+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:14.219 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:14.228+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:14.300 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:14.309+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:15.040 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:15.049+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:15.179 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:15.188+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:15.282 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:15.291+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:15.999 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:16.008+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:16.203 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:16.212+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:16.246 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:16.255+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:17.030 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:17.039+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:17.191 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:17.200+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:17.233 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:17.242+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:17.988 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:17.997+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:18.189 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:18.198+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:18.227 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:18.236+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:19.004 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:19.013+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:19.214 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:19.223+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:19.264 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:19.273+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:20.053 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:20.062+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:20.264 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:20.273+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:20.277 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:20.286+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:21.005 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:21.014+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:21.242 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:21.251+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:21.286 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:21.295+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:21.962 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:21.971+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:22.234 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:22.243+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:22.247 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:22.256+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:22.948 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:22.957+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:23.242 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:23.251+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:23.279 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:23.288+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:23.917 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:23.926+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:24.196 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:24.205+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:24.319 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:24.328+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:24.920 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:24.929+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:25.202 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:25.211+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:25.294 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:25.303+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:25.882 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:25.891+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:26.158 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:26.167+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:26.282 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:26.292+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:26.835 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:26.844+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:27.139 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:27.148+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:27.243 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:27.253+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:27.841 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:27.851+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:28.127 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:28.137+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:28.219 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:28.229+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:28.794 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:28.804+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:29.089 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:29.099+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:29.215 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:29.225+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:29.802 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:29.812+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:30.104 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:30.114+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:30.182 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:30.192+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:30.848 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:30.858+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:31.061 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:31.071+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:31.143 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:31.153+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:31.829 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:31.839+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:32.060 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:32.070+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:32.110 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:32.121+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:32.878 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:32.888+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:33.020 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:33.029+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:33.087 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:33.096+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:33.830 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:33.840+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:34.047 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:34.057+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:34.067 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:34.077+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:34.850 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:34.860+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:35.035 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:35.045+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:35.093 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:35.103+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:35.852 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:35.862+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:36.068 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:36.078+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:36.084 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:36.094+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:36.842 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:36.852+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:37.054 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:37.064+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:37.100 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:37.111+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:37.851 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:37.862+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:38.068 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:38.078+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:38.107 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:38.118+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:38.887 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:38.897+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:39.067 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:39.077+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:39.133 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:39.143+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:39.907 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:39.917+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:40.066 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:40.076+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:40.172 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:40.182+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:40.910 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:40.919+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:41.051 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:41.060+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:41.148 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:41.157+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:41.929 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:41.938+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:42.008 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:42.017+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:42.115 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:42.124+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:42.927 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:42.936+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:43.019 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:43.029+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:43.118 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:43.126+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:43.907 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:43.916+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:44.047 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:44.056+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:44.114 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:44.123+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:44.864 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:44.873+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:45.040 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:45.050+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:45.109 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:45.119+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:45.861 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:45.871+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:46.049 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:46.056+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:46.098 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:46.107+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:46.845 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:46.855+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:47.015 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:47.025+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:47.130 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:47.139+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:47.871 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:47.880+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:47.992 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:48.001+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:48.108 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:48.117+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:48.893 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:48.902+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:48.955 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:48.964+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:49.093 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:49.102+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:49.906 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:49.915+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:49.951 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:49.960+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:50.077 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:50.087+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:50.871 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:50.881+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:50.973 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:50.982+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:51.100 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:51.109+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:51.822 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:51.832+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:51.998 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:52.008+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:52.064 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:52.073+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:52.840 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:52.849+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:53.031 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:53.040+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:53.066 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:53.075+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:53.841 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:53.851+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:53.992 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:54.001+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:54.052 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:54.061+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:54.887 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:54.897+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:55.008 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:55.017+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:55.081 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:55.091+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:55.904 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:55.913+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:56.042 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:56.051+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:56.063 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:56.073+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:56.871 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:56.880+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:57.019 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:57.028+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:57.051 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:57.061+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:57.883 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:57.892+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:58.066 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:58.075+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:58.100 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:58.110+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:58.880 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:58.889+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:59.061 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:35:59.070+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:59.114 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:35:59.124+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:35:59.881 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:35:59.891+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:00.082 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:00.091+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:00.121 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:00.128+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:00.869 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:00.878+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:01.053 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:01.062+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:01.076 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:01.086+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:01.860 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:01.869+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:02.071 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:02.081+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:02.116 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:02.126+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:02.818 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:02.827+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:03.040 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:03.049+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:03.086 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:03.096+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:03.772 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:03.781+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:04.002 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:04.011+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:04.125 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:04.134+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:04.758 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:04.767+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:05.028 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:05.038+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:05.099 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:05.108+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:05.766 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:05.775+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:05.983 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:05.993+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:06.063 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:06.072+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:06.804 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:06.813+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:06.951 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:06.961+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:07.074 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:07.083+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:07.775 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:07.784+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:07.984 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:07.993+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:08.093 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:08.103+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:08.807 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:08.817+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:09.023 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:09.033+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:09.101 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:09.110+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:09.839 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:09.849+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:10.044 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:10.053+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:10.080 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:10.090+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 22 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:10.824 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:10.834+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:11.040 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:11.050+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:11.103 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:11.112+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:11.780 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:11.790+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:11.994 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:12.003+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:12.087 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:12.098+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:12.823 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:12.834+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:13.003 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:13.013+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:13.114 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:13.125+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:13.828 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:13.839+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:14.038 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:14.048+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:14.103 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:14.113+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:14.879 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:14.889+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:15.039 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:15.050+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:15.142 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:15.152+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:15.857 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:15.868+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:16.004 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:16.015+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:16.118 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:16.128+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:16.819 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:16.830+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:16.973 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:16.984+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:17.151 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:17.161+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:17.771 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:17.782+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:18.004 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:18.014+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:18.188 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:18.199+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:18.766 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:18.777+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:18.968 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:18.979+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:19.212 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:19.223+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:19.750 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:19.761+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:20.014 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:20.024+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:20.167 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:20.177+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:20.751 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:20.762+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:20.976 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:20.986+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:21.184 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:21.195+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:21.752 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:21.762+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:22.004 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:22.015+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:22.233 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:22.244+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:22.735 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:22.746+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:22.961 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:22.972+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:23.239 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:23.250+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:23.698 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:23.708+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:23.993 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:24.004+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:24.239 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:24.249+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:24.744 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:24.755+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:24.979 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:24.990+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:25.213 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:25.224+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:25.767 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:25.777+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:25.953 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:25.963+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:26.207 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:26.216+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:26.803 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:26.813+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:27.003 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:27.013+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:27.187 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:27.196+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:27.781 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:27.791+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:27.963 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:27.972+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:28.222 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:28.232+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:28.787 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:28.797+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:28.943 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:28.953+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:29.189 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:29.197+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:29.797 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:29.807+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:29.968 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:29.978+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:30.147 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:30.157+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:30.785 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:30.794+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:30.973 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:30.983+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:31.118 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:31.128+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:31.740 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:31.750+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:32.003 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:32.012+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:32.091 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:32.101+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:32.761 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:32.771+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:32.971 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:32.981+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:33.120 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:33.129+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:33.789 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:33.799+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:33.966 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:33.975+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:34.165 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:34.175+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:34.823 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:34.833+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:34.939 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:34.948+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:35.192 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:35.202+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:35.867 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:35.877+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:35.955 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:35.965+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:36.216 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:36.226+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:36.867 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:36.877+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:36.945 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:36.955+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:37.201 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:37.211+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:37.911 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:37.920+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:37.911 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:37.921+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:38.166 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:38.176+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:38.863 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:38.873+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:38.910 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:38.920+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:39.206 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:39.216+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:39.886 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:39.895+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:39.900 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:39.911+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:40.251 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:40.261+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:40.855 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:40.865+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:40.919 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:40.929+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:41.261 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:41.271+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:41.813 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:41.823+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:41.896 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:41.906+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:42.299 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:42.308+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:42.836 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:42.846+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:42.891 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:42.901+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:43.293 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:43.303+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:43.864 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:43.873+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:43.924 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:43.934+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:44.295 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:44.304+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:44.828 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:44.838+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:44.931 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:44.941+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:45.250 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:45.260+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:45.875 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:45.885+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:45.912 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:45.921+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:46.298 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:46.308+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:46.835 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:46.845+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:46.885 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:46.895+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:47.317 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:47.326+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:47.863 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:47.873+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:47.890 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:47.900+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:48.363 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:48.372+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:48.855 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:48.865+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:48.863 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:48.873+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:49.358 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:49.368+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:49.839 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:49.849+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:49.865 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:49.875+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:50.377 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:50.386+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:50.853 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:50.863+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:50.870 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:50.879+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:51.350 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:51.360+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:51.833 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:51.843+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:51.911 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:51.921+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:52.369 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:52.379+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:52.838 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:52.848+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:52.947 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:52.957+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:53.368 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:53.378+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:53.814 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:53.824+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:53.934 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:53.944+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:54.362 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:54.372+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:54.819 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:54.829+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:54.918 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:54.929+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:55.352 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:55.362+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:55.869 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:55.879+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:55.929 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:55.939+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:56.352 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:56.363+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:56.820 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:56.831+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:56.974 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:56.985+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:57.336 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:57.347+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:57.803 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:57.814+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:57.987 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:57.998+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:58.368 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:58.379+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:58.816 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:58.827+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:59.024 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:36:59.035+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:59.389 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:36:59.400+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:36:59.783 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:36:59.794+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:00.046 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:00.057+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:00.406 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:00.417+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:00.735 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:00.746+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:01.012 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:01.024+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:01.377 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:01.388+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:01.734 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:01.745+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:01.964 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:01.975+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:02.330 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:02.341+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:02.761 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:02.772+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:02.987 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:02.998+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:03.361 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:03.372+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:03.727 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:03.738+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:04.021 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:04.032+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:04.355 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:04.366+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:04.713 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:04.724+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:05.059 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:05.070+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:05.374 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:05.385+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:05.693 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:05.704+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:06.059 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:06.070+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:06.374 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:06.385+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:06.712 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:06.723+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:07.071 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:07.082+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:07.416 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:07.427+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:07.703 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:07.714+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:08.043 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:08.054+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:08.368 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:08.378+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:08.719 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:08.729+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:09.066 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:09.078+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:09.337 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:09.348+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:09.679 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:09.690+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:10.095 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:10.106+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:10.345 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:10.356+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:10.690 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:10.700+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:11.138 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:11.148+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:11.337 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:11.347+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:11.732 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:11.742+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:12.125 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:12.135+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:12.362 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:12.372+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:12.706 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:12.716+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:13.090 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:13.100+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:13.330 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:13.340+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:13.691 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:13.701+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:14.094 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:14.104+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:14.342 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:14.352+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:14.733 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:14.743+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:15.142 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:15.152+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:15.299 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:15.309+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:15.697 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:15.707+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:16.121 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:16.128+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:16.322 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:16.332+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:16.651 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:16.661+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:17.097 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:17.107+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:17.296 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:17.306+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:17.619 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:17.629+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:18.136 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:18.146+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:18.342 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:18.352+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:18.655 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:18.665+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:19.123 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:19.133+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:19.379 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:19.389+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:19.626 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:19.636+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:20.097 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:20.107+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:20.360 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:20.370+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:20.591 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:20.601+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:21.130 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:21.140+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:21.364 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:21.374+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:21.548 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:21.558+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:22.124 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:22.134+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:22.324 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:22.334+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:22.542 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:22.553+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:23.166 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:23.176+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:23.310 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:23.320+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:23.579 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:23.590+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:24.193 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:24.203+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:24.322 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:24.332+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:24.618 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:24.628+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:25.178 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:25.188+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:25.371 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:25.382+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:25.660 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:25.670+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:26.225 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:26.235+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:26.413 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:26.424+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:26.680 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:26.690+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:27.274 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:27.285+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:27.438 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:27.448+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:27.704 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:27.714+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:28.237 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:28.247+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:28.458 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:28.468+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:28.678 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:28.688+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:29.267 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:29.277+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:29.429 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:29.439+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:29.724 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:29.735+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:30.315 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:30.325+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:30.389 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:30.399+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:30.754 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:30.764+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:31.301 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:31.311+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:31.353 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:31.363+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:31.783 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:31.794+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:32.253 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:32.263+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:32.402 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:32.413+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:32.818 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:32.829+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:33.289 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:33.300+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:33.388 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:33.399+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:33.780 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:33.790+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:34.280 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:34.291+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:34.360 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:34.370+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:34.810 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:34.820+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:35.313 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:35.323+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:35.375 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:35.385+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:35.800 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:35.810+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:36.323 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:36.334+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:36.328 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:36.338+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:36.758 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:36.768+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:37.335 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:37.345+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:37.350 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:37.361+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:37.782 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:37.793+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:38.345 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:38.355+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:38.380 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:38.390+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:38.785 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:38.795+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:39.332 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:39.342+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:39.350 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:39.356+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:39.814 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:39.824+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:40.357 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:40.367+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:40.388 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:40.400+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:40.798 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:40.808+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:41.370 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:41.380+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:41.384 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:41.394+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:41.770 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:41.780+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:42.335 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:42.346+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:42.374 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:42.385+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:42.800 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:42.811+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:43.352 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:43.363+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:43.358 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:43.369+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:43.849 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:43.860+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:44.349 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:44.360+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:44.388 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:44.399+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:44.876 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:44.887+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:45.306 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:45.318+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:45.343 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:45.355+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:45.881 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:45.893+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:46.288 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:46.299+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:46.388 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:46.399+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:46.914 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:46.925+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:47.258 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:47.269+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:47.429 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:47.440+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:47.899 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:47.910+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:48.277 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:48.289+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:48.397 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:48.408+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:48.949 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:48.961+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:49.257 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:49.268+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:49.372 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:49.383+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:49.947 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:49.959+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:50.287 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:50.299+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:50.396 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:50.408+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:50.928 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:50.939+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:51.283 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:51.295+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:51.390 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:51.402+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:51.952 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:51.962+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:52.248 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:52.260+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:52.352 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:52.363+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:52.912 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:52.923+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:53.218 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:53.229+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:53.350 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:53.361+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:53.922 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:53.933+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:54.215 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:54.227+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:54.381 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:54.393+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:54.951 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:54.963+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:55.254 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:55.265+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:55.349 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:55.359+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:55.975 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:55.986+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:56.265 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:56.276+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:56.329 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:56.339+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:56.999 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:57.009+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:57.234 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:57.244+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:57.302 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:57.313+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:57.984 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:57.994+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:58.197 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:58.208+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:58.308 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:58.318+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:58.961 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:37:58.972+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:59.213 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:37:59.224+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:37:59.260 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:37:59.271+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:00.010 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:00.020+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:00.227 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:00.238+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:00.288 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:00.299+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:01.025 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:01.035+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:01.200 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:01.210+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:01.244 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:01.254+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:02.017 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:02.028+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:02.211 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:02.221+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:02.213 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:02.223+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:03.045 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:03.056+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:03.181 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:03.192+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:03.255 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:03.266+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:04.075 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:04.085+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:04.187 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:04.198+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:04.241 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:04.251+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:05.066 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:05.077+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:05.211 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:05.215+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:05.212 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:05.217+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:06.114 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:06.125+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:06.179 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:06.189+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:06.254 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:06.265+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:07.150 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:07.160+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:07.166 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:07.176+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:07.252 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:07.263+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:08.138 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:08.148+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:08.187 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:08.198+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:08.268 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:08.278+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:09.105 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:09.115+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:09.157 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:09.165+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:09.291 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:09.301+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:10.110 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:10.121+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:10.169 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:10.180+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:10.292 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:10.303+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:11.125 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:11.135+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:11.209 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:11.220+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:11.265 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:11.275+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:12.143 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:12.153+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:12.217 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:12.228+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:12.274 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:12.285+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:13.154 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:13.164+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:13.222 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:13.233+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:13.247 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:13.258+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:14.189 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:14.199+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:14.253 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:14.264+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:14.279 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:14.290+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:15.190 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:15.200+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:15.215 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:15.226+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:15.263 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:15.273+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:16.167 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:16.178+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:16.220 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:16.230+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:16.288 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:16.299+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:17.182 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:17.192+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:17.183 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:17.194+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:17.259 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:17.269+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:18.190 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:18.201+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:18.226 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:18.236+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:18.236 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:18.246+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:19.195 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:19.206+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:19.247 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:19.258+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:19.261 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:19.272+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:20.172 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:20.183+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:20.263 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:20.274+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:20.296 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:20.307+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:21.184 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:21.194+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:21.290 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:21.300+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:21.336 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:21.347+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:22.222 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:22.233+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:22.251 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:22.262+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:22.289 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:22.300+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:23.191 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:23.202+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:23.258 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:23.268+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:23.297 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:23.308+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:24.188 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:24.199+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:24.249 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:24.260+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:24.278 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:24.289+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:25.140 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:25.151+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:25.258 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:25.269+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:25.258 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:25.269+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:26.190 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:26.201+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:26.235 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:26.244+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:26.307 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:26.318+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:27.225 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:27.236+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:27.238 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:27.250+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:27.267 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:27.278+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:28.221 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:28.232+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:28.259 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:28.271+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:28.278 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:28.290+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:29.202 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:29.214+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:29.256 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:29.268+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:29.284 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:29.296+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:30.178 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:30.190+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:30.216 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:30.228+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:30.246 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:30.258+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:31.189 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:31.201+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:31.247 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:31.258+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:31.284 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:31.296+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:32.181 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:32.192+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:32.232 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:32.244+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:32.273 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:32.284+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:33.148 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:33.160+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:33.233 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:33.245+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:33.258 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:33.270+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:34.191 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:34.203+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:34.193 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:34.206+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:34.299 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:34.311+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:35.169 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:35.181+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:35.171 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:35.183+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:35.323 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:35.335+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:36.142 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:36.154+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:36.169 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:36.181+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:36.276 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:36.288+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:37.173 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:37.185+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:37.185 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:37.197+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:37.310 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:37.322+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:38.137 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:38.148+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:38.156 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:38.168+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:38.268 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:38.280+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:39.156 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:39.168+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:39.181 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:39.192+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:39.227 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:39.239+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:40.178 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:40.190+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:40.210 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:40.222+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:40.211 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:40.223+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:41.139 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:41.150+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:41.206 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:41.217+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:41.224 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:41.235+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:42.093 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:42.104+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:42.179 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:42.189+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:42.183 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:42.194+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:43.046 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:43.057+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:43.162 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:43.172+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:43.227 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:43.238+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:44.085 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:44.095+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:44.149 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:44.160+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:44.213 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:44.224+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:45.051 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:45.062+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:45.161 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:45.172+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:45.221 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:45.232+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:46.009 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:46.020+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:46.163 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:46.174+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:46.199 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:46.209+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:47.003 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:47.014+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:47.125 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:47.136+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:47.237 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:47.248+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:47.990 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:48.001+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:48.158 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:48.169+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:48.252 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:48.263+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:48.966 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:48.977+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:49.154 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:49.165+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:49.236 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:49.247+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:49.945 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:49.956+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:50.151 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:50.162+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:50.216 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:50.227+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:50.928 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:50.939+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:51.200 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:51.210+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:51.253 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:51.264+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:51.904 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:51.915+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:52.182 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:52.193+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:52.287 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:52.298+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:52.893 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:52.904+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:53.157 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:53.168+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:53.282 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:53.293+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:53.918 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:53.928+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:54.115 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:54.126+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:54.308 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:54.319+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:54.945 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:54.956+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:55.070 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:55.081+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:55.286 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:55.297+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:55.938 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:55.949+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:56.045 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:56.056+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:56.255 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:56.266+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:56.974 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:56.985+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:57.093 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:57.104+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:57.223 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:57.233+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:57.985 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:57.996+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:58.119 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:58.130+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:58.199 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:58.210+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:58.981 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:58.992+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:59.156 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:38:59.167+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:59.160 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:38:59.171+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:38:59.969 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:38:59.980+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:00.114 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:00.125+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:00.195 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:00.206+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:00.997 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:01.008+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:01.081 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:01.092+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:01.215 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:01.226+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:01.965 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:01.976+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:02.051 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:02.062+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:02.223 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:02.234+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:02.998 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:03.009+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:03.037 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:03.048+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:03.246 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:03.257+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:03.955 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:03.966+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:04.000 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:04.011+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:04.289 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:04.300+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:04.984 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:04.995+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:04.999 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:05.010+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:05.254 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:05.265+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:05.938 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:05.949+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:05.992 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:06.003+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:06.303 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:06.315+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:06.963 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:06.974+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:06.969 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:06.981+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:07.323 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:07.335+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:07.957 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:07.968+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:07.977 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:07.988+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:08.342 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:08.353+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:08.972 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:08.983+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:08.979 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:08.990+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:09.335 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:09.346+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:09.924 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:09.936+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:09.992 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:10.003+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:10.318 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:10.329+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:10.907 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:10.918+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:11.041 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:11.052+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:11.271 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:11.282+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:11.876 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:11.887+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:12.061 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:12.073+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:12.296 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:12.308+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:12.906 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:12.918+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:13.031 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:13.043+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:13.312 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:13.324+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:13.878 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:13.890+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:14.006 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:14.018+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:14.286 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:14.299+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:14.838 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:14.850+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:15.044 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:15.056+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:15.315 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:15.327+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:15.806 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:15.818+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:16.024 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:16.036+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:16.307 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:16.319+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:16.800 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:16.813+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:17.032 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:17.045+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:17.278 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:17.291+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:17.777 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:17.789+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:18.051 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:18.063+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:18.253 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:18.266+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:18.827 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:18.839+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:19.027 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:19.039+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:19.279 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:19.292+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:19.869 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:19.881+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:20.036 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:20.048+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:20.256 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:20.268+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:20.847 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:20.859+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:21.059 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:21.071+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:21.230 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:21.242+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:21.854 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:21.866+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:22.017 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:22.029+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:22.185 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:22.197+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:22.838 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:22.850+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:23.045 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:23.057+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:23.215 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:23.227+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:23.832 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:23.844+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:24.092 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:24.104+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:24.176 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:24.188+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:24.855 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:24.867+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:25.060 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:25.072+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:25.135 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:25.147+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:25.838 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:25.850+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:26.035 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:26.046+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:26.161 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:26.172+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:26.869 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:26.880+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:26.996 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:27.007+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:27.116 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:27.127+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:27.858 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:27.869+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:28.023 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:28.034+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:28.120 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:28.131+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:28.906 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:28.917+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:29.024 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:29.035+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:29.162 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:29.173+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:29.861 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:29.872+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:29.979 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:29.990+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:30.171 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:30.182+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:30.909 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:30.921+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:31.000 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:31.011+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:31.147 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:31.159+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:31.947 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:31.958+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:31.999 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:32.010+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:32.184 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:32.195+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:32.915 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:32.926+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:33.003 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:33.014+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:33.202 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:33.214+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:33.954 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:33.965+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:34.036 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:34.048+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:34.213 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:34.220+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:34.952 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:34.963+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:34.990 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:35.001+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:35.259 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:35.271+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:35.940 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:35.952+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:35.983 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:35.994+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:36.296 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:36.308+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:36.936 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:36.947+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:36.973 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:36.985+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:37.322 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:37.333+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:37.951 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:37.962+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:38.002 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:38.013+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:38.286 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:38.298+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:38.947 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:38.958+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:38.969 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:38.981+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:39.238 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:39.250+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:39.906 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:39.918+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:39.963 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:39.975+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:40.233 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:40.244+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:40.875 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:40.886+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:41.013 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:41.024+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:41.243 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:41.254+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:41.903 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:41.914+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:41.987 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:41.998+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:42.284 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:42.296+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:42.913 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:42.925+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:43.006 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:43.017+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:43.303 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:43.315+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:43.926 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:43.938+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:44.037 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:44.048+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:44.321 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:44.332+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:44.888 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:44.900+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:45.013 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:45.025+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:45.356 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:45.368+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:45.843 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:45.854+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:46.039 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:46.051+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:46.359 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:46.371+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:46.820 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:46.832+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:47.072 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:47.083+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:47.410 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:47.421+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:47.772 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:47.784+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:48.122 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:48.134+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:48.407 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:48.419+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:48.772 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:48.783+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:49.112 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:49.124+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:49.399 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:49.411+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:49.771 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:49.782+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:50.069 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:50.079+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:50.449 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:50.460+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:50.756 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:50.768+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:51.109 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:51.117+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:51.402 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:51.414+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:51.711 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:51.723+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:52.141 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:52.150+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:52.358 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:52.369+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:52.708 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:52.719+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:53.118 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:53.130+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:53.329 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:53.340+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:53.708 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:53.720+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:54.127 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:54.139+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:54.331 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:54.342+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:54.692 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:54.703+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:55.086 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:55.099+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:55.287 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:55.299+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:55.655 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:55.666+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:56.137 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:56.148+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:56.245 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:56.256+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:56.644 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:56.655+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:57.173 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:57.185+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:57.219 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:57.231+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:57.623 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:57.636+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:58.192 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:58.205+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:58.248 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:58.260+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:58.605 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:58.617+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:59.142 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:39:59.155+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:59.232 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:39:59.244+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:39:59.583 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:39:59.595+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:00.112 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:00.125+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:00.184 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:00.197+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:00.573 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:00.586+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:01.068 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:01.080+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:01.154 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:01.166+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:01.555 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:01.567+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:02.048 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:02.061+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:02.124 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:02.136+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:02.564 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:02.576+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:03.090 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:03.102+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:03.167 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:03.180+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:03.522 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:03.534+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:04.106 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:04.119+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:04.211 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:04.224+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:04.499 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:04.512+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:05.154 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:05.167+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:05.213 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:05.226+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:05.475 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:05.488+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:06.203 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:06.216+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:06.249 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:06.262+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:06.462 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:06.475+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:07.210 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:07.223+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:07.266 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:07.279+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:07.441 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:07.454+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:08.242 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:08.253+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:08.255 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:08.268+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:08.449 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:08.461+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:09.290 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:09.303+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:09.305 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:09.318+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:09.481 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:09.494+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:10.294 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:10.306+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:10.302 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:10.314+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:10.461 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:10.473+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:11.258 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:11.269+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:11.305 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:11.317+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:11.426 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:11.438+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:12.303 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:12.315+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:12.310 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:12.322+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:12.387 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:12.398+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:13.296 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:13.308+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:13.329 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:13.340+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:13.401 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:13.413+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:14.272 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:14.284+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:14.352 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:14.363+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:14.408 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:14.420+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:15.250 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:15.262+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:15.317 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:15.329+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:15.431 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:15.442+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:16.210 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:16.222+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:16.286 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:16.298+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:16.439 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:16.450+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:17.196 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:17.207+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:17.262 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:17.273+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:17.457 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:17.468+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:18.197 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:18.208+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:18.232 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:18.244+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:18.458 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:18.469+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:19.156 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:19.168+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:19.205 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:19.217+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:19.430 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:19.442+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:20.159 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:20.171+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:20.198 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:20.210+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:20.417 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:20.429+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:21.176 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:21.188+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:21.235 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:21.247+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:21.421 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:21.433+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:22.193 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:22.205+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:22.243 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:22.255+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:22.377 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:22.389+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:23.157 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:23.169+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:23.221 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:23.232+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:23.390 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:23.402+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:24.147 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:24.158+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:24.254 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:24.266+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:24.356 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:24.367+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:25.144 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:25.156+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:25.214 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:25.226+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:25.381 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:25.392+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:26.106 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:26.118+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:26.201 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:26.213+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:26.400 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:26.412+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:27.099 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:27.111+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:27.157 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:27.169+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:27.377 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:27.389+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:28.101 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:28.112+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:28.146 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:28.158+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:28.424 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:28.436+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:29.106 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:29.118+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:29.119 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:29.131+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:29.415 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:29.427+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:30.107 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:30.119+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:30.123 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:30.135+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:30.410 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:30.422+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:31.093 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:31.105+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:31.148 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:31.160+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:31.424 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:31.435+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:32.116 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:32.127+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:32.179 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:32.191+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:32.439 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:32.451+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:33.115 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:33.127+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:33.222 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:33.233+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:33.479 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:33.490+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:34.130 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:34.142+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:34.227 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:34.239+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:34.503 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:34.514+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:35.084 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:35.095+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:35.242 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:35.254+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:35.535 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:35.547+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:36.105 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:36.116+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:36.195 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:36.207+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:36.530 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:36.542+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:37.146 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:37.158+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:37.208 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:37.220+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:37.533 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:37.545+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:38.184 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:38.195+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:38.190 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:38.202+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:38.500 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:38.512+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:39.218 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:39.230+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:39.219 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:39.231+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:39.541 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:39.553+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:40.227 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:40.239+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:40.240 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:40.251+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:40.499 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:40.511+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:41.192 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:41.204+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:41.275 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:41.286+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:41.462 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:41.475+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:42.231 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:42.242+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:42.232 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:42.245+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:42.480 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:42.493+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:43.245 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:43.257+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:43.276 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:43.289+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:43.466 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:43.478+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:44.218 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:44.231+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:44.234 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:44.247+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:44.462 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:44.475+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:45.230 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:45.243+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:45.269 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:45.282+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:45.504 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:45.517+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:46.275 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:46.288+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:46.309 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:46.322+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:46.498 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:46.511+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:47.275 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:47.288+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:47.285 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:47.298+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:47.536 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:47.549+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:48.228 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:48.240+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:48.317 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:48.330+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:48.547 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:48.560+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:49.197 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:49.210+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:49.326 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:49.339+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:49.539 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:49.552+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:50.240 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:50.252+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:50.335 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:50.348+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:50.556 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:50.569+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:51.263 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:51.275+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:51.356 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:51.369+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:51.592 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:51.605+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:52.258 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:52.271+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:52.380 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:52.393+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:52.554 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:52.567+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:53.300 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:53.313+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:53.369 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:53.382+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:53.599 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:53.612+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:54.288 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:54.300+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:54.394 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:54.407+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:54.602 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:54.615+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:55.310 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:55.322+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:55.445 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:55.457+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:55.558 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:55.570+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:56.330 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:56.341+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:56.483 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:56.495+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:56.584 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:56.596+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:57.295 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:57.307+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:57.461 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:57.473+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:57.541 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:57.552+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:58.281 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:58.293+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:58.437 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:58.448+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:58.506 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:58.518+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:59.294 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:40:59.306+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:59.478 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:40:59.490+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:40:59.541 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:40:59.553+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:00.258 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:00.270+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:00.509 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:00.520+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:00.549 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:00.561+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:01.305 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:01.317+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:01.498 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:01.509+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:01.578 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:01.590+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:02.260 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:02.272+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:02.496 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:02.508+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:02.553 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:02.565+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:03.261 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:03.273+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:03.500 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:03.512+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:03.604 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:03.616+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:04.218 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:04.230+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:04.482 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:04.494+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:04.570 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:04.582+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:05.227 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:05.239+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:05.495 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:05.507+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:05.553 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:05.565+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:06.258 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:06.270+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:06.466 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:06.479+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:06.590 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:06.602+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:07.209 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:07.221+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:07.475 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:07.487+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:07.606 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:07.617+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:08.175 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:08.187+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:08.442 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:08.454+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:08.632 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:08.644+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:09.223 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:09.235+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:09.403 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:09.416+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:09.626 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:09.638+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:10.192 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:10.204+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:10.367 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:10.379+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:10.640 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:10.652+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:11.155 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:11.167+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:11.388 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:11.400+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:11.631 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:11.643+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:12.121 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:12.133+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:12.416 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:12.428+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:12.650 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:12.662+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:13.164 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:13.175+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:13.402 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:13.414+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:13.670 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:13.682+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:14.161 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:14.173+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:14.377 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:14.389+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:14.671 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:14.683+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:15.137 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:15.149+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:15.350 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:15.362+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:15.686 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:15.698+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:16.168 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:16.180+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:16.324 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:16.336+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:16.734 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:16.745+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:17.191 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:17.203+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:17.326 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:17.338+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:17.736 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:17.748+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:18.238 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:18.250+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:18.281 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:18.293+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:18.753 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:18.765+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:19.236 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:19.248+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:19.267 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:19.279+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:19.728 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:19.740+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:20.188 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:20.200+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:20.233 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:20.245+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:20.758 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:20.770+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:21.168 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:21.180+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:21.283 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:21.295+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:21.791 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:21.803+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:22.141 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:22.153+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:22.279 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:22.291+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:22.829 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:22.841+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:23.112 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:23.124+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:23.273 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:23.285+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:23.873 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:23.885+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:24.096 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:24.108+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:24.278 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:24.290+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:24.848 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:24.860+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:25.125 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:25.137+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:25.312 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:25.325+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:25.829 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:25.842+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:26.078 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:26.090+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:26.294 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:26.306+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:26.866 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:26.878+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:27.054 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:27.066+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:27.291 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:27.304+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:27.834 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:27.847+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:28.024 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:28.037+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:28.295 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:28.308+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:28.803 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:28.816+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:29.033 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:29.046+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:29.305 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:29.318+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:29.755 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:29.768+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:30.050 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:30.063+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:30.343 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:30.356+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:30.735 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:30.749+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:31.016 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:31.029+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:31.303 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:31.316+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:31.687 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:31.701+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:31.979 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:31.992+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:32.339 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:32.352+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:32.706 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:32.719+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:32.961 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:32.974+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:33.367 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:33.380+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:33.745 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:33.758+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:33.929 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:33.942+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:34.331 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:34.344+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:34.715 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:34.728+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:34.891 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:34.905+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:35.346 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:35.359+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:35.738 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:35.751+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:35.929 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:35.942+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:36.367 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:36.380+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:36.721 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:36.734+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:36.949 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:36.961+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:37.363 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:37.376+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:37.758 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:37.771+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:37.927 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:37.941+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:38.377 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:38.391+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:38.720 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:38.733+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:38.959 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:38.972+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:39.367 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:39.380+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:39.735 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:39.748+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:39.973 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:39.986+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:40.325 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:40.337+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:40.706 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:40.718+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:40.981 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:40.993+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:41.286 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:41.298+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:41.718 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:41.730+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:41.958 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:41.970+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:42.279 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:42.291+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:42.696 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:42.708+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:42.970 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:42.982+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:43.257 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:43.269+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:43.682 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:43.694+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:43.969 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:43.982+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:44.227 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:44.239+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:44.637 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:44.650+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:45.009 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:45.021+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:45.216 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:45.229+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:45.679 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:45.692+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:45.981 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:45.994+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:46.241 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:46.253+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:46.714 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:46.726+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:47.016 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:47.028+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:47.220 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:47.232+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:47.712 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:47.724+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:47.984 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:47.996+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:48.182 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:48.195+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:48.741 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:48.750+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:49.031 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:49.044+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:49.218 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:49.231+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:49.758 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:49.770+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:50.064 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:50.076+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:50.217 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:50.229+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:50.733 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:50.745+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:51.024 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:51.036+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:51.185 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:51.197+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:51.702 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:51.714+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:52.060 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:52.072+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:52.168 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:52.180+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:52.726 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:52.737+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:53.014 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:53.026+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:53.218 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:53.230+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:53.700 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:53.712+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:53.969 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:53.982+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:54.181 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:54.193+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:54.748 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:54.761+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:54.929 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:54.942+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:55.184 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:55.196+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:55.765 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:55.777+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:55.935 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:55.947+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:56.205 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:56.218+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:56.809 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:56.820+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:56.891 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:56.904+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:57.184 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:57.196+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:57.854 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:57.866+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:57.896 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:57.908+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:58.159 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:58.172+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:58.850 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:58.863+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:58.891 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:58.904+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:59.154 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:41:59.166+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:59.808 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:41:59.821+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:41:59.854 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:41:59.866+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:00.109 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:00.121+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:00.847 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:00.860+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:00.891 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:00.904+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:01.143 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:01.153+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:01.881 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:01.893+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:01.939 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:01.951+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:02.146 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:02.159+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:02.869 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:02.882+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:02.918 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:02.931+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:03.107 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:03.119+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:03.856 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:03.866+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:03.962 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:03.974+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:04.089 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:04.101+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:04.829 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:04.842+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:04.945 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:04.958+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:05.114 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:05.126+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:05.823 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:05.836+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:05.939 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:05.951+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:06.098 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:06.111+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:06.869 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:06.881+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:06.932 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:06.944+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:07.139 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:07.150+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:07.874 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:07.887+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:07.964 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:07.976+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:08.136 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:08.149+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:08.888 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:08.900+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:08.957 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:08.969+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:09.091 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:09.104+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:09.870 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:09.882+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:09.934 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:09.946+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:10.116 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:10.128+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:10.888 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:10.900+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:10.920 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:10.932+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:11.144 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:11.157+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:11.843 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:11.855+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:11.958 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:11.970+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:12.160 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:12.174+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:12.797 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:12.811+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:12.986 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:12.998+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:13.142 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:13.155+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:13.830 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:13.843+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:13.981 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:13.995+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:14.177 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:14.190+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:14.859 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:14.873+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:14.975 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:14.989+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:15.136 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:15.149+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:15.819 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:15.833+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:15.961 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:15.975+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:16.091 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:16.104+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:16.825 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:16.838+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:16.926 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:16.939+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:17.042 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:17.056+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:17.842 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:17.855+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:17.882 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:17.895+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:18.021 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:18.034+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:18.843 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:18.856+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:18.882 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:18.895+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:19.061 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:19.074+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:19.830 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:19.843+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:19.913 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:19.926+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:20.033 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:20.046+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:20.869 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:20.883+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:20.910 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:20.923+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:21.006 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:21.019+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:21.896 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:21.909+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:21.942 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:21.955+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:22.043 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:22.056+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:22.849 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:22.863+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:22.955 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:22.969+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:23.038 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:23.052+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:23.821 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:23.835+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:23.911 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:23.924+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:24.057 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:24.070+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:24.847 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:24.860+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:24.900 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:24.914+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:25.070 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:25.083+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:25.831 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:25.843+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:25.942 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:25.954+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:26.024 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:26.036+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:26.853 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:26.865+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:26.918 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:26.930+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:27.067 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:27.079+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:27.885 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:27.897+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:27.890 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:27.903+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:28.067 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:28.080+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:28.884 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:28.896+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:28.905 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:28.917+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:29.055 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:29.067+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:29.836 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:29.848+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:29.886 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:29.899+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:30.097 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:30.110+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:30.849 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:30.862+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:30.878 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:30.891+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:31.123 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:31.136+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:31.835 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:31.847+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:31.927 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:31.940+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:32.097 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:32.107+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:32.818 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:32.831+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:32.900 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:32.912+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:33.130 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:33.142+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:33.831 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:33.844+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:33.900 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:33.913+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:34.166 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:34.176+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:34.801 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:34.813+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:34.874 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:34.886+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:35.169 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:35.180+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:35.823 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:35.835+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:35.851 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:35.863+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:36.152 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:36.165+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:36.853 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:36.865+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:36.901 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:36.914+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:37.142 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:37.155+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:37.830 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:37.843+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:37.928 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:37.941+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:38.161 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:38.174+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:38.811 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:38.824+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:38.961 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:38.960+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:39.152 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:39.165+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:39.771 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:39.784+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:39.937 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:39.949+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:40.139 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:40.152+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:40.763 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:40.776+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:40.946 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:40.958+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:41.133 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:41.146+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:41.734 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:41.747+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:41.962 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:41.974+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:42.126 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:42.139+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:42.737 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:42.748+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:42.972 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:42.985+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:43.098 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:43.111+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:43.767 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:43.780+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:43.972 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:43.985+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:44.050 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:44.062+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:44.800 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:44.813+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:44.957 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:44.970+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:45.098 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:45.111+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:45.840 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:45.853+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:45.983 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:45.995+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:46.130 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:46.142+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:46.888 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:46.901+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:46.999 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:47.012+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:47.089 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:47.102+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:47.843 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:47.856+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:48.036 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:48.048+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:48.090 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:48.103+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:48.887 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:48.899+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:49.008 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:49.021+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:49.058 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:49.070+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:49.929 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:49.942+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:50.011 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:50.024+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:50.027 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:50.039+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:50.969 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:50.981+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:50.992 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:51.005+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:51.007 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:51.020+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:51.977 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:51.989+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:52.012 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:52.025+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:52.041 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:52.054+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:53.004 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:53.016+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:53.020 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:53.033+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:53.059 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:53.072+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:54.022 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:54.035+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:54.039 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:54.051+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:54.104 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:54.117+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:55.013 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:55.026+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:55.060 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:55.072+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:55.148 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:55.161+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:55.994 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:56.007+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:56.020 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:56.033+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:56.126 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:56.139+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:57.015 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:57.028+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:57.064 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:57.078+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:57.151 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:57.165+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:58.008 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:58.022+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:58.106 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:58.120+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:58.108 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:58.122+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:58.968 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:58.982+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:59.113 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:42:59.126+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:59.150 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:42:59.161+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:42:59.978 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:42:59.990+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:00.145 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:00.159+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:00.186 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:00.200+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:01.025 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:01.039+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:01.143 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:01.157+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:01.147 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:01.161+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:02.010 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:02.024+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:02.131 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:02.144+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:02.196 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:02.210+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:03.047 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:03.061+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:03.161 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:03.175+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:03.228 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:03.242+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:04.067 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:04.081+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:04.169 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:04.183+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:04.214 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:04.227+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:05.066 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:05.080+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:05.162 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:05.175+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:05.193 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:05.206+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:06.084 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:06.097+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:06.137 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:06.151+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:06.208 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:06.222+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:07.085 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:07.099+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:07.148 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:07.162+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:07.180 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:07.194+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:08.046 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:08.060+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:08.142 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:08.156+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:08.185 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:08.199+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:09.067 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:09.080+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:09.140 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:09.153+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:09.159 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:09.173+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:10.083 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:10.097+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:10.091 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:10.105+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:10.146 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:10.159+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:11.051 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:11.064+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:11.137 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:11.150+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:11.153 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:11.166+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:12.055 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:12.068+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:12.126 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:12.139+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:12.138 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:12.151+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:13.050 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:13.062+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:13.088 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:13.101+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:13.187 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:13.200+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:14.004 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:14.017+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:14.114 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:14.127+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:14.237 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:14.250+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:15.024 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:15.036+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:15.095 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:15.108+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:15.285 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:15.297+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:16.030 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:16.043+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:16.078 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:16.091+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:16.281 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:16.294+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:16.999 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:17.011+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:17.095 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:17.108+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:17.286 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:17.298+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:18.000 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:18.013+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:18.057 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:18.070+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:18.328 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:18.341+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:18.993 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:19.006+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:19.038 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:19.050+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:19.328 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:19.341+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:19.991 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:20.004+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:20.036 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:20.049+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:20.350 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:20.363+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:21.023 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:21.035+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:21.024 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:21.037+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:21.308 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:21.321+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:22.025 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:22.037+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:22.039 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:22.052+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:22.278 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:22.291+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:22.998 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:23.011+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:23.067 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:23.080+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:23.327 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:23.340+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:24.039 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:24.052+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:24.083 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:24.096+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:24.315 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:24.328+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:25.081 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:25.094+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:25.115 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:25.128+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:25.350 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:25.362+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:26.091 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:26.104+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:26.142 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:26.155+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:26.361 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:26.371+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:27.054 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:27.067+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:27.101 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:27.114+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:27.321 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:27.334+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:28.050 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:28.063+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:28.066 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:28.079+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:28.351 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:28.364+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:29.076 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:29.089+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:29.096 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:29.109+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:29.358 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:29.370+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:30.054 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:30.066+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:30.098 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:30.111+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:30.363 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:30.376+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:31.028 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:31.041+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:31.103 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:31.115+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:31.403 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:31.415+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:32.071 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:32.084+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:32.109 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:32.121+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:32.363 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:32.375+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:33.063 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:33.076+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:33.131 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:33.144+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:33.339 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:33.352+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:34.026 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:34.039+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:34.154 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:34.167+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:34.359 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:34.372+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:34.991 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:35.004+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:35.145 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:35.158+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:35.324 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:35.337+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:36.003 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:36.016+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:36.099 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:36.112+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:36.350 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:36.363+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:37.005 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:37.018+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:37.107 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:37.120+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:37.306 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:37.319+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:37.960 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:37.973+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:38.101 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:38.114+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:38.268 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:38.281+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:38.929 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:38.942+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:39.118 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:39.130+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:39.260 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:39.274+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:39.908 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:39.921+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:40.099 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:40.112+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:40.292 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:40.305+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:40.907 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:40.918+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:41.086 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:41.100+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:41.268 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:41.282+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:41.867 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:41.880+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:42.071 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:42.086+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:42.265 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:42.279+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:42.868 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:42.881+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:43.100 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:43.114+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:43.272 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:43.286+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:43.858 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:43.872+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:44.053 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:44.067+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:44.280 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:44.294+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:44.893 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:44.907+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:45.026 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:45.040+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:45.258 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:45.271+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:45.920 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:45.934+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:46.056 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:46.070+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:46.246 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:46.260+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:46.899 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:46.913+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:47.037 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:47.051+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:47.287 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:47.301+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:47.950 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:47.964+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:48.031 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:48.045+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:48.332 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:48.346+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:48.950 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:48.964+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:49.036 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:49.050+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:49.288 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:49.302+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:49.970 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:49.984+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:50.077 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:50.091+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:50.295 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:50.309+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:50.966 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:50.980+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:51.072 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:51.086+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:51.247 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:51.261+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:51.995 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:52.008+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:52.080 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:52.094+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:52.296 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:52.310+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:52.972 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:52.986+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:53.081 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:53.095+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:53.254 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:53.268+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:53.946 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:53.960+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:54.109 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:54.123+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:54.298 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:54.312+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:54.920 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:54.934+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:55.137 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:55.151+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:55.316 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:55.330+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:55.956 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:55.969+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:56.119 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:56.132+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:56.289 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:56.302+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:56.913 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:56.926+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:57.132 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:57.145+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:57.243 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:57.256+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:57.902 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:57.915+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:58.175 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:58.188+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:58.247 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:58.260+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:58.938 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:58.952+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:59.202 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:43:59.215+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:59.276 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:43:59.289+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:43:59.933 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:43:59.946+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:00.175 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:00.188+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:00.303 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:00.316+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:00.968 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:00.981+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:01.150 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:01.164+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:01.261 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:01.274+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:01.928 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:01.942+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:02.125 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:02.138+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:02.234 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:02.248+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:02.958 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:02.971+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:03.138 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:03.151+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:03.278 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:03.291+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:03.995 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:04.008+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:04.156 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:04.169+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:04.233 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:04.246+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:04.959 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:04.972+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:05.187 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:05.200+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:05.250 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:05.263+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:06.001 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:06.014+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:06.150 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:06.163+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:06.239 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:06.253+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:07.031 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:07.044+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:07.159 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:07.172+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:07.226 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:07.239+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:08.014 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:08.027+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:08.192 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:08.205+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:08.240 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:08.253+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:08.991 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:09.004+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:09.172 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:09.183+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:09.228 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:09.241+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:09.968 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:09.981+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:10.154 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:10.167+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:10.233 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:10.246+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:10.961 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:10.974+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:11.161 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:11.174+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:11.236 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:11.249+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:11.951 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:11.964+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:12.149 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:12.162+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:12.216 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:12.229+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:12.925 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:12.938+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:13.132 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:13.145+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:13.214 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:13.226+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:13.901 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:13.914+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:14.111 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:14.124+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:14.191 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:14.204+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:14.900 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:14.913+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:15.075 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:15.088+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:15.227 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:15.240+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:15.946 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:15.959+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:16.059 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:16.072+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:16.208 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:16.221+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:16.915 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:16.927+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:17.062 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:17.075+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:17.209 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:17.222+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:17.904 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:17.917+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:18.028 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:18.041+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:18.176 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:18.189+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:18.880 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:18.893+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:19.033 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:19.046+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:19.173 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:19.186+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:19.892 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:19.906+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:20.027 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:20.041+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:20.151 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:20.164+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:20.856 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:20.869+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:21.004 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:21.017+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:21.128 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:21.142+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:21.888 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:21.901+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:21.971 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:21.985+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:22.103 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:22.117+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:22.890 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:22.903+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:22.957 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:22.971+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:23.111 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:23.124+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:23.902 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:23.915+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:23.939 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:23.952+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:24.109 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:24.122+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:24.855 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:24.868+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:24.989 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:25.003+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:25.109 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:25.123+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:25.864 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:25.878+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:25.975 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:25.988+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:26.103 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:26.116+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:26.900 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:26.913+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:27.000 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:27.013+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:27.127 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:27.141+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:27.937 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:27.951+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:28.004 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:28.019+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:28.149 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:28.163+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:28.930 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:28.944+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:28.995 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:29.010+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:29.112 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:29.126+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:29.925 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:29.939+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:30.031 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:30.045+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:30.124 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:30.139+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:30.970 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:30.985+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:31.053 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:31.067+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:31.103 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:31.117+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:32.017 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:32.031+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:32.091 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:32.106+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:32.092 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:32.106+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:32.981 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:32.995+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:33.076 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:33.090+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:33.088 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:33.103+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:34.014 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:34.028+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:34.029 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:34.044+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:34.111 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:34.125+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:35.050 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:35.064+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:35.056 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:35.071+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:35.079 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:35.094+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:36.042 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:36.056+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:36.050 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:36.064+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:36.069 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:36.083+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:37.030 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:37.044+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:37.063 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:37.077+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:37.086 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:37.101+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:38.026 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:38.040+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:38.031 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:38.045+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:38.124 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:38.138+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:38.989 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:39.003+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:39.038 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:39.052+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:39.089 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:39.104+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:39.989 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:40.003+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:40.022 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:40.036+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:40.051 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:40.065+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:40.964 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:40.978+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:40.995 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:41.008+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:41.063 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:41.077+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:41.933 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:41.947+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:41.967 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:41.980+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:42.098 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:42.111+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:42.951 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:42.964+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:42.997 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:43.011+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:43.133 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:43.147+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:43.957 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:43.970+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:43.972 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:43.985+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:44.116 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:44.130+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:44.953 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:44.966+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:44.998 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:45.012+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:45.163 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:45.177+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:45.908 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:45.921+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:46.034 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:46.048+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:46.173 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:46.187+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:46.865 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:46.878+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:47.028 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:47.041+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:47.128 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:47.141+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:47.881 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:47.895+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:47.995 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:48.008+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:48.082 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:48.095+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:48.903 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:48.917+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:48.948 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:48.961+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:49.116 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:49.130+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:49.931 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:49.944+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:49.953 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:49.966+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:50.097 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:50.111+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:50.976 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:50.989+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:50.988 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:51.002+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:51.124 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:51.138+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:52.006 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:52.019+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:52.022 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:52.035+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:52.175 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:52.188+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:52.991 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:53.005+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:53.072 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:53.086+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:53.150 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:53.164+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:54.011 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:54.025+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:54.113 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:54.126+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:54.198 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:54.212+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:54.965 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:54.978+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:55.085 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:55.098+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:55.164 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:55.178+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:55.941 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:55.954+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:56.100 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:56.113+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:56.179 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:56.193+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:56.921 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:56.935+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:57.055 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:57.068+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:57.180 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:57.194+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:57.933 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:57.947+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:58.067 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:58.081+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:58.222 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:58.235+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:58.918 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:58.931+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:59.050 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:44:59.064+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:59.220 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:44:59.234+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:44:59.884 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:44:59.897+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:00.067 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:00.080+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:00.246 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:00.259+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:00.901 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:00.915+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:01.036 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:01.050+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:01.209 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:01.223+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:01.939 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:01.952+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:02.045 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:02.059+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:02.170 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:02.180+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:02.949 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:02.962+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:03.054 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:03.067+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:03.142 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:03.156+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:03.969 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:03.982+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:04.089 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:04.102+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:04.145 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:04.158+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:05.009 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:05.023+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:05.049 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:05.062+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:05.115 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:05.129+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:05.967 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:05.980+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:06.013 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:06.027+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:06.153 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:06.167+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:06.988 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:07.001+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:07.055 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:07.068+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:07.188 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:07.202+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:07.973 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:07.987+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:08.058 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:08.072+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:08.144 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:08.157+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:08.955 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:08.969+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:09.097 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:09.110+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:09.108 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:09.122+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:09.942 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:09.956+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:10.069 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:10.082+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:10.157 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:10.171+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:10.982 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:10.994+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:11.099 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:11.113+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:11.127 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:11.140+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:11.952 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:11.966+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:12.108 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:12.123+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:12.138 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:12.153+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:12.984 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:12.998+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:13.114 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:13.128+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:13.136 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:13.150+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:13.976 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:13.990+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:14.106 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:14.121+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:14.155 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:14.169+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:15.002 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:15.016+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:15.119 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:15.133+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:15.164 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:15.179+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:16.023 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:16.037+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:16.080 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:16.095+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:16.142 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:16.156+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:17.056 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:17.070+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:17.077 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:17.091+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:17.110 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:17.124+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:18.042 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:18.056+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:18.050 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:18.065+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:18.094 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:18.108+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:19.001 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:19.015+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:19.088 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:19.103+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:19.144 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:19.159+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:20.034 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:20.048+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:20.122 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:20.137+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:20.152 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:20.166+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:21.082 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:21.097+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:21.156 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:21.171+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:21.169 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:21.183+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:22.071 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:22.085+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:22.162 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:22.177+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:22.171 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:22.186+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:23.065 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:23.079+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:23.131 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:23.145+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:23.198 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:23.213+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:24.022 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:24.037+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:24.121 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:24.135+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:24.200 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:24.214+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:24.981 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:24.994+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:25.118 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:25.132+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:25.187 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:25.200+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:26.027 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:26.041+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:26.083 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:26.096+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:26.142 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:26.156+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:26.996 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:27.010+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:27.120 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:27.133+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:27.133 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:27.146+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:27.992 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:28.006+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:28.123 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:28.136+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:28.160 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:28.173+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:28.973 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:28.987+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:29.109 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:29.122+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:29.171 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:29.185+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:29.942 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:29.953+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:30.103 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:30.116+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:30.156 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:30.170+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:30.902 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:30.915+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:31.126 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:31.140+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:31.170 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:31.184+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:31.857 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:31.871+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:32.124 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:32.138+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:32.148 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:32.162+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:32.892 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:32.906+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:33.112 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:33.125+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:33.188 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:33.202+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:33.924 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:33.937+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:34.129 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:34.142+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:34.169 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:34.183+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:34.887 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:34.901+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:35.126 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:35.139+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:35.156 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:35.169+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:35.873 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:35.887+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:36.150 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:36.164+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:36.164 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:36.178+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:36.865 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:36.879+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:37.125 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:37.138+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:37.182 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:37.195+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:37.898 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:37.912+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:38.077 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:38.091+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:38.163 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:38.177+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:38.891 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:38.904+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:39.050 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:39.064+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:39.124 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:39.138+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:39.862 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:39.876+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:40.091 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:40.105+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:40.130 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:40.141+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:40.879 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:40.893+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:41.075 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:41.089+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:41.119 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:41.133+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:41.887 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:41.900+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:42.104 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:42.117+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:42.131 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:42.145+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:42.899 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:42.912+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:43.084 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:43.097+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:43.114 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:43.128+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:43.942 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:43.956+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:44.065 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:44.079+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:44.111 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:44.125+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:44.942 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:44.956+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:45.077 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:45.088+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:45.102 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:45.116+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:45.987 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:46.001+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:46.073 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:46.087+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:46.145 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:46.159+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:47.010 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:47.023+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:47.101 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:47.114+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:47.179 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:47.193+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:47.971 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:47.985+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:48.146 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:48.160+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:48.221 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:48.235+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:48.976 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:48.990+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:49.132 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:49.146+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:49.241 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:49.255+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:49.977 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:49.990+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:50.159 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:50.173+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:50.236 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:50.250+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:50.988 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:51.002+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:51.190 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:51.204+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:51.242 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:51.255+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:52.033 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:52.047+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:52.156 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:52.170+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:52.245 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:52.258+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:53.050 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:53.064+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:53.151 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:53.164+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:53.196 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:53.210+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:54.048 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:54.061+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:54.175 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:54.189+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:54.189 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:54.202+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:55.016 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:55.030+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:55.158 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:55.172+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:55.168 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:55.182+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:56.008 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:56.022+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:56.149 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:56.161+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:56.195 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:56.208+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:56.976 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:56.990+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:57.124 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:57.139+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:57.237 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:57.252+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:57.990 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:58.005+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:58.157 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:58.172+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:58.242 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:58.257+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:58.986 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:59.001+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:59.180 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:45:59.195+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:59.264 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:45:59.278+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:45:59.940 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:45:59.955+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:00.140 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:00.154+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:00.298 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:00.313+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:00.984 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:00.999+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:01.179 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:01.194+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:01.343 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:01.352+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:01.951 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:01.966+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:02.209 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:02.224+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:02.363 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:02.378+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:02.916 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:02.931+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:03.237 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:03.252+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:03.340 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:03.355+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:03.890 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:03.905+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:04.280 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:04.295+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:04.305 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:04.320+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:04.867 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:04.882+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:05.273 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:05.288+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:05.310 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:05.325+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:05.854 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:05.868+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:06.243 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:06.258+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:06.283 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:06.298+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:06.850 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:06.863+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:07.280 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:07.294+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:07.332 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:07.347+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:07.872 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:07.886+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:08.281 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:08.296+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:08.355 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:08.370+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:08.834 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:08.849+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:09.275 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:09.290+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:09.317 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:09.332+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:09.868 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:09.883+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:10.234 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:10.248+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:10.302 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:10.313+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:10.880 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:10.894+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:11.195 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:11.209+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:11.258 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:11.272+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:11.891 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:11.905+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:12.213 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:12.227+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:12.214 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:12.228+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:12.916 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:12.930+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:13.204 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:13.218+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:13.244 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:13.257+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:13.964 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:13.978+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:14.199 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:14.213+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:14.284 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:14.298+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:14.984 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:14.998+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:15.248 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:15.262+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:15.332 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:15.346+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:16.012 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:16.026+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:16.269 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:16.283+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:16.358 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:16.371+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:17.027 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:17.041+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:17.248 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:17.262+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:17.375 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:17.389+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:17.993 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:18.007+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:18.210 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:18.224+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:18.416 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:18.430+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:19.025 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:19.039+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:19.251 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:19.265+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:19.429 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:19.442+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:20.030 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:20.043+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:20.284 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:20.298+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:20.443 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:20.457+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:21.073 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:21.087+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:21.247 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:21.261+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:21.395 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:21.409+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:22.101 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:22.115+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:22.257 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:22.271+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:22.352 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:22.366+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:23.139 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:23.153+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:23.235 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:23.249+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:23.329 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:23.343+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:24.146 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:24.160+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:24.285 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:24.299+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:24.373 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:24.387+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:25.122 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:25.136+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:25.263 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:25.277+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:25.361 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:25.375+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:26.135 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:26.148+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:26.313 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:26.327+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:26.389 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:26.402+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:27.103 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:27.116+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:27.315 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:27.329+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:27.397 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:27.411+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:28.097 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:28.110+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:28.345 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:28.358+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:28.385 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:28.399+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:29.092 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:29.105+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:29.357 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:29.371+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:29.399 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:29.413+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:30.109 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:30.123+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:30.343 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:30.356+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:30.442 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:30.456+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:31.090 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:31.104+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:31.303 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:31.317+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:31.437 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:31.451+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:32.075 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:32.089+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:32.288 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:32.302+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:32.465 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:32.479+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:33.063 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:33.077+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:33.310 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:33.324+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:33.433 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:33.447+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:34.031 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:34.044+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:34.327 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:34.341+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:34.480 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:34.494+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:35.017 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:35.031+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:35.305 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:35.319+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:35.477 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:35.491+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:36.060 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:36.074+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:36.267 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:36.281+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:36.479 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:36.493+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:37.012 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:37.025+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:37.250 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:37.264+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:37.481 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:37.495+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:37.963 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:37.977+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:38.214 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:38.228+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:38.515 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:38.529+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:39.012 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:39.026+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:39.247 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:39.261+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:39.490 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:39.504+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:40.016 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:40.030+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:40.221 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:40.235+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:40.466 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:40.480+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:41.041 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:41.055+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:41.172 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:41.186+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:41.505 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:41.519+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:42.036 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:42.051+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:42.131 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:42.146+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:42.523 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:42.538+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:43.045 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:43.060+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:43.128 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:43.143+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:43.552 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:43.567+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:44.023 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:44.038+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:44.178 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:44.193+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:44.557 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:44.572+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:45.069 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:45.083+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:45.172 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:45.187+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:45.549 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:45.561+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:46.038 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:46.052+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:46.153 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:46.168+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:46.524 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:46.539+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:47.002 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:47.016+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:47.151 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:47.166+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:47.506 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:47.521+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:47.988 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:48.003+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:48.140 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:48.155+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:48.524 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:48.539+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:48.949 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:48.964+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:49.138 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:49.153+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:49.556 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:49.571+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:49.902 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:49.917+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:50.142 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:50.157+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:50.532 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:50.547+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:50.915 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:50.930+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:51.171 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:51.183+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:51.500 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:51.515+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:51.942 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:51.957+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:52.120 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:52.135+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:52.461 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:52.476+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:52.913 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:52.928+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:53.089 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:53.103+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:53.439 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:53.454+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:53.903 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:53.918+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:54.074 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:54.089+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:54.475 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:54.489+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:54.943 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:54.958+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:55.032 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:55.046+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:55.486 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:55.501+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:55.984 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:55.998+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:56.071 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:56.085+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:56.524 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:56.538+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:57.002 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:57.016+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:57.032 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:57.046+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:57.554 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:57.568+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:57.964 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:57.978+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:57.998 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:58.012+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:58.545 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:58.559+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:58.947 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:58.961+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:59.046 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:46:59.060+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:59.524 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:46:59.538+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:46:59.950 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:46:59.964+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:00.027 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:00.041+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:00.558 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:00.572+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:00.967 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:00.981+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:01.024 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:01.038+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:01.527 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:01.541+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:02.007 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:02.021+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:02.063 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:02.078+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:02.502 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:02.516+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:02.989 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:03.003+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:03.091 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:03.105+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:03.514 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:03.528+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:03.946 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:03.960+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:04.059 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:04.073+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:04.489 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:04.504+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:04.983 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:04.997+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:05.024 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:05.038+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:05.528 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:05.542+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:06.017 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:06.030+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:06.050 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:06.064+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:06.504 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:06.518+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:06.985 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:06.999+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:07.035 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:07.049+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:07.496 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:07.510+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:07.980 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:07.994+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:08.031 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:08.045+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:08.461 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:08.475+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:09.015 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:09.029+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:09.027 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:09.041+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:09.429 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:09.443+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:09.983 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:09.998+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:10.026 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:10.040+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:10.405 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:10.419+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:10.936 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:10.950+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:11.052 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:11.066+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:11.403 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:11.417+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:11.951 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:11.966+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:12.091 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:12.105+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:12.397 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:12.411+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:12.944 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:12.958+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:13.101 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:13.115+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:13.361 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:13.375+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:13.965 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:13.979+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:14.107 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:14.121+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:14.325 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:14.339+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:14.919 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:14.933+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:15.080 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:15.094+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:15.365 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:15.379+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:15.924 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:15.938+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:16.118 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:16.132+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:16.376 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:16.390+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:16.910 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:16.924+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:17.112 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:17.126+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:17.385 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:17.399+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:17.944 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:17.959+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:18.103 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:18.118+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:18.395 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:18.410+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:18.910 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:18.924+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:19.061 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:19.075+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:19.436 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:19.449+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:19.937 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:19.951+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:20.015 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:20.029+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:20.434 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:20.448+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:20.892 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:20.906+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:21.022 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:21.036+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:21.389 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:21.403+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:21.861 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:21.875+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:22.039 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:22.054+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:22.439 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:22.454+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:22.817 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:22.831+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:23.077 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:23.092+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:23.454 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:23.468+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:23.824 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:23.838+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:24.122 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:24.136+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:24.478 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:24.492+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:24.775 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:24.790+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:25.128 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:25.142+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:25.448 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:25.463+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:25.759 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:25.774+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:26.179 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:26.193+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:26.418 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:26.432+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:26.741 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:26.754+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:27.130 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:27.146+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:27.429 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:27.444+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:27.724 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:27.739+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:28.086 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:28.101+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:28.389 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:28.404+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:28.676 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:28.691+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:29.131 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:29.146+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:29.376 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:29.391+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:29.718 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:29.733+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:30.109 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:30.124+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:30.386 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:30.402+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:30.762 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:30.777+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:31.128 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:31.143+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:31.422 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:31.438+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:31.804 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:31.819+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:32.123 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:32.138+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:32.432 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:32.447+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:32.779 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:32.795+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:33.128 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:33.144+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:33.474 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:33.489+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:33.818 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:33.833+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:34.150 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:34.165+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:34.459 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:34.474+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:34.792 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:34.807+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:35.102 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:35.117+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:35.464 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:35.479+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:35.805 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:35.820+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:36.100 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:36.115+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:36.467 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:36.482+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:36.816 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:36.832+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:37.117 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:37.133+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:37.445 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:37.460+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:37.847 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:37.862+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:38.097 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:38.112+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:38.458 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:38.473+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:38.882 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:38.897+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:39.084 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:39.099+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:39.441 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:39.456+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:39.927 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:39.941+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:40.039 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:40.053+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:40.464 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:40.478+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:40.969 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:40.983+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:41.053 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:41.067+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:41.429 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:41.443+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:41.951 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:41.965+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:42.051 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:42.065+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:42.466 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:42.481+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:42.997 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:43.011+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:43.018 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:43.032+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:43.428 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:43.443+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:44.012 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:44.026+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:44.050 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:44.064+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:44.439 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:44.453+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:45.004 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:45.019+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:45.054 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:45.068+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:45.418 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:45.431+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:45.967 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:45.982+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:46.077 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:46.092+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:46.433 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:46.447+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:46.988 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:47.002+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:47.068 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:47.082+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:47.399 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:47.414+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:47.999 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:48.014+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:48.064 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:48.078+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:48.380 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:48.394+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:48.994 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:49.008+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:49.016 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:49.031+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:49.368 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:49.382+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:49.963 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:49.977+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:49.990 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:50.005+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:50.410 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:50.424+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:50.917 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:50.931+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:50.962 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:50.977+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:51.409 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:51.423+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:51.889 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:51.903+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:51.932 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:51.946+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:52.366 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:52.380+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:52.857 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:52.871+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:52.913 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:52.927+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:53.384 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:53.399+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:53.872 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:53.886+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:53.919 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:53.933+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:54.404 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:54.418+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:54.837 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:54.852+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:54.941 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:54.955+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:55.410 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:55.424+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:55.797 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:55.811+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:55.933 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:55.948+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:56.429 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:56.443+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:56.827 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:56.842+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:56.966 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:56.980+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:57.431 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:57.445+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:57.865 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:57.880+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:57.920 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:57.935+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:58.424 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:58.438+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:58.825 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:58.839+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:58.913 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:58.928+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:59.461 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:47:59.475+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:59.843 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:47:59.858+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:47:59.926 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:47:59.940+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:00.457 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:00.471+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:00.815 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:00.830+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:00.975 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:00.989+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:01.485 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:01.500+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:01.806 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:01.820+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:02.014 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:02.028+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:02.470 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:02.485+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:02.766 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:02.781+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:03.023 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:03.037+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:03.459 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:03.474+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:03.756 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:03.771+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:04.047 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:04.062+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:04.468 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:04.483+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:04.772 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:04.787+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:05.008 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:05.022+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:05.433 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:05.447+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:05.779 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:05.793+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:06.032 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:06.046+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:06.387 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:06.401+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:06.827 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:06.842+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:07.041 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:07.055+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:07.419 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:07.433+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:07.839 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:07.853+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:08.086 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:08.101+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:08.468 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:08.483+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:08.799 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:08.814+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:09.089 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:09.102+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:09.493 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:09.507+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:09.788 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:09.802+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:10.088 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:10.102+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:10.514 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:10.528+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:10.766 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:10.781+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:11.142 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:11.152+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:11.521 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:11.536+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:11.780 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:11.796+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:12.112 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:12.127+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:12.519 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:12.535+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:12.779 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:12.795+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:13.102 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:13.117+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:13.549 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:13.564+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:13.769 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:13.785+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:14.113 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:14.129+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:14.580 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:14.595+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:14.758 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:14.774+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:15.077 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:15.092+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:15.552 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:15.568+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:15.751 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:15.767+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:16.028 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:16.044+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:16.594 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:16.610+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:16.799 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:16.814+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:17.049 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:17.064+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:17.558 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:17.572+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:17.794 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:17.809+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:18.092 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:18.107+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:18.591 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:18.607+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:18.805 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:18.820+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:19.063 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:19.079+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:19.630 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:19.645+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:19.785 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:19.801+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:20.074 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:20.089+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:20.654 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:20.670+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:20.814 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:20.830+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:21.028 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:21.043+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:21.681 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:21.696+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:21.787 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:21.797+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:22.063 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:22.078+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:22.684 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:22.700+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:22.749 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:22.765+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:23.058 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:23.073+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:23.692 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:23.708+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:23.781 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:23.796+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:24.055 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:24.070+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:24.718 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:24.733+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:24.791 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:24.806+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:25.060 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:25.075+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:25.756 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:25.771+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:25.789 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:25.804+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:26.070 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:26.085+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:26.748 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:26.762+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:26.781 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:26.796+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:27.033 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:27.048+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:27.785 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:27.800+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:27.787 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:27.802+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:28.048 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:28.063+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:28.753 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:28.767+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:28.827 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:28.841+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:29.013 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:29.028+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:29.761 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:29.776+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:29.826 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:29.840+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:29.989 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:30.003+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:30.742 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:30.757+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:30.796 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:30.810+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:31.036 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:31.050+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:31.703 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:31.717+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:31.758 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:31.772+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:32.040 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:32.055+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:32.736 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:32.750+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:32.746 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:32.760+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:33.025 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:33.039+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:33.710 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:33.725+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:33.781 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:33.795+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:34.018 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:34.033+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:34.714 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:34.729+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:34.825 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:34.839+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:35.068 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:35.083+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:35.697 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:35.711+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:35.819 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:35.833+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:36.024 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:36.038+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:36.719 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:36.733+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:36.815 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:36.830+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:36.990 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:37.004+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:37.733 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:37.747+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:37.793 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:37.808+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:37.990 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:38.005+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:38.750 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:38.765+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:38.798 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:38.812+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:38.992 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:39.007+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:39.728 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:39.742+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:39.843 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:39.857+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:39.977 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:39.992+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:40.771 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:40.785+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:40.841 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:40.855+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:40.939 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:40.953+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:41.730 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:41.745+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:41.822 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:41.837+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:41.924 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:41.938+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:42.713 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:42.727+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:42.810 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:42.823+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:42.968 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:42.983+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:43.735 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:43.749+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:43.760 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:43.775+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:43.954 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:43.969+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:44.721 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:44.735+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:44.774 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:44.789+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:44.957 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:44.971+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:45.722 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:45.736+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:45.807 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:45.822+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:46.006 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:46.021+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:46.694 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:46.709+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:46.796 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:46.811+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:46.978 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:46.993+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:47.689 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:47.704+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:47.763 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:47.777+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:47.981 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:47.996+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:48.717 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:48.731+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:48.736 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:48.751+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:48.973 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:48.988+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:49.694 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:49.709+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:49.740 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:49.755+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:50.000 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:50.014+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:50.652 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:50.667+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:50.736 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:50.751+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:50.992 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:51.006+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:51.658 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:51.673+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:51.727 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:51.741+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:51.993 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:52.007+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:52.682 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:52.696+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:52.685 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:52.700+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:52.997 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:53.011+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:53.682 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:53.697+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:53.701 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:53.716+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:54.027 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:54.041+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:54.688 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:54.703+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:54.702 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:54.717+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:55.063 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:55.078+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:55.683 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:55.698+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:55.704 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:55.718+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:56.032 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:56.047+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:56.706 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:56.722+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:56.724 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:56.740+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:56.988 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:57.003+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:57.709 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:57.724+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:57.757 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:57.770+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:57.965 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:57.978+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:58.724 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:58.740+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:58.743 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:58.759+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:58.962 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:58.978+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:59.701 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:48:59.717+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:59.720 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:48:59.736+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:48:59.957 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:48:59.972+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:00.676 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:00.692+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:00.761 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:00.777+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:00.999 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:01.014+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:01.663 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:01.678+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:01.810 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:01.826+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:02.010 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:02.026+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:02.624 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:02.639+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:02.854 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:02.870+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:03.029 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:03.044+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:03.653 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:03.668+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:03.846 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:03.862+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:03.995 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:04.010+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:04.627 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:04.642+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:04.869 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:04.884+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:04.997 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:05.013+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:05.590 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:05.605+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:05.898 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:05.914+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:05.978 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:05.993+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:06.546 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:06.561+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:06.932 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:06.948+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:06.992 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:07.008+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:07.593 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:07.608+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:07.913 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:07.928+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:07.982 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:07.997+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:08.597 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:08.612+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:08.941 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:08.957+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:08.984 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:09.000+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:09.647 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:09.662+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:09.926 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:09.941+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:09.951 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:09.966+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:10.621 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:10.636+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:10.964 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:10.978+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:10.976 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:10.991+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:11.595 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:11.610+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:11.935 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:11.950+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:11.957 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:11.971+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:12.617 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:12.629+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:12.938 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:12.952+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:12.982 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:12.996+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:13.646 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:13.661+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:13.909 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:13.924+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:13.956 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:13.970+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:14.697 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:14.712+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:14.925 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:14.940+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:15.007 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:15.021+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:15.697 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:15.712+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:15.889 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:15.904+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:15.961 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:15.976+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:16.680 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:16.695+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:16.878 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:16.893+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:16.964 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:16.979+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:17.701 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:17.716+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:17.923 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:17.938+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:18.001 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:18.016+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:18.749 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:18.764+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:18.972 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:18.986+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:18.991 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:19.006+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:19.789 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:19.803+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:19.926 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:19.941+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:20.010 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:20.025+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:20.828 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:20.843+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:20.876 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:20.891+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:20.998 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:21.013+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:21.858 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:21.872+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:21.894 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:21.909+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:22.007 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:22.022+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:22.889 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:22.904+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:22.943 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:22.958+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:23.055 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:23.070+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:23.909 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:23.923+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:23.927 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:23.941+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:24.093 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:24.108+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:24.883 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:24.898+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:24.928 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:24.943+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:25.137 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:25.152+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:25.865 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:25.879+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:25.899 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:25.914+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:26.131 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:26.146+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:26.866 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:26.881+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:26.866 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:26.881+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:27.177 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:27.192+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:27.819 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:27.833+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:27.837 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:27.851+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:28.217 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:28.232+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:28.837 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:28.851+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:28.870 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:28.885+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:29.232 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:29.247+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:29.822 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:29.837+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:29.885 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:29.900+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:30.270 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:30.285+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:30.831 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:30.846+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:30.844 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:30.857+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:31.281 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:31.296+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:31.847 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:31.862+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:31.858 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:31.872+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:32.305 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:32.320+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:32.819 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:32.834+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:32.824 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:32.839+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:33.286 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:33.301+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:33.794 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:33.809+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:33.832 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:33.847+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:34.321 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:34.335+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:34.766 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:34.781+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:34.874 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:34.888+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:35.296 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:35.311+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:35.719 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:35.733+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:35.893 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:35.908+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:36.296 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:36.311+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:36.718 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:36.733+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:36.851 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:36.866+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:37.281 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:37.296+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:37.742 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:37.757+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:37.901 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:37.916+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:38.248 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:38.263+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:38.760 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:38.774+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:38.943 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:38.958+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:39.232 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:39.246+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:39.747 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:39.762+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:39.904 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:39.919+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:40.260 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:40.275+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:40.776 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:40.792+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:40.911 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:40.926+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:41.291 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:41.306+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:41.745 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:41.761+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:41.946 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:41.962+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:42.274 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:42.290+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:42.750 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:42.766+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:42.945 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:42.961+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:43.246 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:43.262+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:43.724 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:43.740+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:43.978 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:43.994+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:44.283 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:44.299+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:44.680 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:44.696+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:44.930 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:44.946+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:45.247 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:45.263+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:45.681 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:45.697+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:45.940 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:45.956+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:46.260 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:46.276+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:46.658 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:46.674+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:46.963 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:46.979+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:47.269 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:47.285+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:47.626 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:47.642+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:48.010 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:48.026+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:48.234 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:48.250+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:48.668 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:48.683+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:49.030 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:49.046+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:49.269 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:49.285+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:49.620 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:49.636+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:50.080 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:50.096+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:50.274 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:50.290+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:50.574 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:50.589+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:51.085 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:51.101+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:51.286 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:51.302+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:51.577 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:51.592+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:52.052 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:52.068+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:52.316 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:52.332+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:52.528 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:52.544+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:53.051 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:53.067+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:53.331 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:53.347+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:53.480 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:53.496+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:54.078 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:54.094+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:54.370 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:54.386+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:54.469 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:54.485+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:55.114 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:55.129+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:55.403 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:55.418+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:55.445 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:55.459+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:56.159 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:56.173+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:56.362 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:56.377+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:56.493 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:56.508+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:57.115 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:57.130+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:57.388 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:57.403+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:57.445 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:57.459+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:58.162 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:58.177+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:58.379 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:58.394+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:58.408 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:58.423+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:59.193 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:49:59.208+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:59.364 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:49:59.379+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:49:59.402 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:49:59.417+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:00.202 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:00.217+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:00.392 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:00.407+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:00.432 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:00.447+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:01.225 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:01.240+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:01.398 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:01.413+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:01.474 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:01.489+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:02.212 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:02.227+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:02.428 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:02.443+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:02.508 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:02.523+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:03.236 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:03.251+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:03.446 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:03.461+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:03.506 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:03.521+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:04.252 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:04.267+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:04.452 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:04.467+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:04.556 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:04.571+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:05.269 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:05.284+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:05.491 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:05.506+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:05.549 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:05.564+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:06.315 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:06.330+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:06.449 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:06.464+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:06.507 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:06.522+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:07.347 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:07.362+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:07.427 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:07.442+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:07.507 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:07.522+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:08.355 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:08.370+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:08.392 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:08.407+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:08.555 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:08.570+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:09.376 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:09.391+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:09.384 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:09.399+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:09.587 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:09.603+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:10.408 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:10.423+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:10.430 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:10.445+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:10.560 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:10.575+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:11.436 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:11.450+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:11.472 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:11.486+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:11.597 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:11.612+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:12.448 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:12.463+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:12.510 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:12.525+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:12.584 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:12.599+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:13.487 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:13.502+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:13.543 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:13.558+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:13.611 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:13.626+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:14.444 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:14.459+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:14.520 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:14.535+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:14.639 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:14.654+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:15.462 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:15.477+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:15.561 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:15.576+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:15.603 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:15.618+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:16.510 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:16.525+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:16.553 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:16.568+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:16.589 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:16.604+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:17.468 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:17.483+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:17.518 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:17.533+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:17.565 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:17.580+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:18.487 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:18.502+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:18.517 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:18.532+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:18.548 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:18.563+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:19.471 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:19.486+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:19.531 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:19.546+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:19.569 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:19.584+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:20.510 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:20.525+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:20.564 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:20.579+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:20.596 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:20.611+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:21.514 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:21.529+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:21.567 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:21.582+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:21.598 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:21.613+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:22.526 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:22.542+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:22.563 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:22.578+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:22.604 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:22.619+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:23.530 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:23.545+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:23.558 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:23.573+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:23.608 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:23.623+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:24.566 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:24.581+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:24.580 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:24.595+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:24.590 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:24.606+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:25.583 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:25.598+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:25.588 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:25.604+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:25.626 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:25.641+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:26.543 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:26.559+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:26.567 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:26.583+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:26.649 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:26.665+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:27.522 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:27.537+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:27.547 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:27.563+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:27.661 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:27.677+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:28.484 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:28.500+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:28.589 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:28.605+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:28.679 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:28.695+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:29.529 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:29.545+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:29.621 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:29.637+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:29.647 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:29.663+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:30.520 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:30.536+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:30.599 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:30.615+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:30.657 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:30.673+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:31.480 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:31.497+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:31.592 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:31.608+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:31.703 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:31.719+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:32.471 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:32.487+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:32.642 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:32.657+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:32.703 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:32.719+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:33.517 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:33.533+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:33.674 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:33.690+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:33.701 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:33.717+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:34.503 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:34.519+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:34.634 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:34.650+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:34.669 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:34.685+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:35.504 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:35.520+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:35.640 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:35.656+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:35.680 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:35.696+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:36.548 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:36.564+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:36.635 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:36.651+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:36.731 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:36.747+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:37.571 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:37.587+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:37.647 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:37.663+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:37.683 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:37.699+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:38.586 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:38.602+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:38.626 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:38.642+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:38.671 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:38.687+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:39.562 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:39.577+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:39.611 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:39.626+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:39.708 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:39.723+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:40.541 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:40.556+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:40.627 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:40.642+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:40.701 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:40.716+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:41.544 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:41.559+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:41.652 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:41.667+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:41.655 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:41.670+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:42.584 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:42.599+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:42.626 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:42.641+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:42.642 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:42.657+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:43.621 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:43.636+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:43.636 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:43.652+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:43.688 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:43.704+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:44.615 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:44.630+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:44.645 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:44.660+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:44.701 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:44.715+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:45.618 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:45.633+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:45.654 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:45.669+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:45.656 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:45.672+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:46.619 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:46.634+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:46.620 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:46.635+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:46.665 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:46.680+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:47.622 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:47.638+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:47.659 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:47.674+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:47.677 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:47.692+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:48.579 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:48.594+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:48.663 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:48.678+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:48.715 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:48.730+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:49.550 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:49.565+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:49.670 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:49.685+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:49.747 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:49.762+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:50.526 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:50.541+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:50.697 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:50.712+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:50.745 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:50.760+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:51.574 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:51.589+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:51.721 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:51.736+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:51.739 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:51.754+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:52.556 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:52.571+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:52.680 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:52.696+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:52.723 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:52.738+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:53.513 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:53.528+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:53.654 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:53.669+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:53.715 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:53.730+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:54.496 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:54.512+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:54.631 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:54.646+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:54.681 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:54.696+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:55.487 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:55.502+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:55.600 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:55.615+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:55.632 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:55.648+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:56.482 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:56.498+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:56.570 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:56.585+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:56.620 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:56.636+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:57.518 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:57.533+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:57.574 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:57.589+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:57.653 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:57.668+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:58.519 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:58.534+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:58.605 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:58.620+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:58.621 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:58.636+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:59.558 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:50:59.574+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:59.602 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:50:59.617+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:50:59.649 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:50:59.664+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:00.512 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:00.528+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:00.605 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:00.621+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:00.626 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:00.641+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:01.554 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:01.570+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:01.569 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:01.584+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:01.611 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:01.627+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:02.550 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:02.566+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:02.554 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:02.570+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:02.586 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:02.601+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:03.517 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:03.533+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:03.543 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:03.558+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:03.600 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:03.615+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:04.491 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:04.506+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:04.573 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:04.588+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:04.650 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:04.665+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:05.513 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:05.528+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:05.598 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:05.613+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:05.645 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:05.660+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:06.524 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:06.539+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:06.549 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:06.564+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:06.599 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:06.614+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:07.523 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:07.538+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:07.555 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:07.570+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:07.575 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:07.590+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:08.479 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:08.494+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:08.519 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:08.535+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:08.558 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:08.574+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:09.440 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:09.455+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:09.497 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:09.512+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:09.559 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:09.574+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:10.416 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:10.431+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:10.517 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:10.532+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 24 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:10.557 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:10.572+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:11.447 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:11.464+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:11.470 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:11.486+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:11.574 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:11.590+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:12.416 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:12.432+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:12.514 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:12.531+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:12.576 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:12.593+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:13.398 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:13.414+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:13.490 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:13.506+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:13.622 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:13.638+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:14.399 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:14.415+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:14.471 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:14.487+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:14.667 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:14.683+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:15.364 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:15.378+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:15.449 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:15.466+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:15.679 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:15.695+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:16.392 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:16.409+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:16.444 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:16.460+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:16.678 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:16.694+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:17.411 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:17.428+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:17.426 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:17.442+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:17.673 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:17.689+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:18.428 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:18.444+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:18.450 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:18.467+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:18.660 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:18.676+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:19.411 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:19.427+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:19.458 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:19.474+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:19.693 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:19.709+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:20.433 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:20.449+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:20.459 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:20.475+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:20.686 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:20.703+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:21.446 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:21.462+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:21.499 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:21.515+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:21.705 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:21.722+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:22.453 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:22.469+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:22.474 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:22.490+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:22.682 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:22.699+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:23.480 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:23.496+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:23.492 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:23.509+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:23.698 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:23.714+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:24.487 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:24.502+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:24.523 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:24.538+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:24.727 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:24.742+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:25.485 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:25.500+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:25.501 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:25.516+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:25.735 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:25.750+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:26.459 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:26.474+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:26.479 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:26.495+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:26.713 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:26.728+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:27.471 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:27.486+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:27.512 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:27.527+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:27.755 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:27.771+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:28.449 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:28.464+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:28.548 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:28.563+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:28.796 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:28.812+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:29.422 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:29.437+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:29.566 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:29.582+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:29.758 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:29.774+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:30.465 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:30.480+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:30.520 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:30.536+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:30.732 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:30.748+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:31.443 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:31.458+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:31.512 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:31.528+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:31.685 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:31.700+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:32.439 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:32.455+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:32.539 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:32.554+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:32.721 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:32.736+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:33.473 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:33.488+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:33.494 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:33.510+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:33.695 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:33.710+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:34.437 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:34.452+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:34.464 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:34.479+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:34.655 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:34.671+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:35.424 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:35.439+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:35.460 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:35.476+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:35.653 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:35.668+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:36.433 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:36.448+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:36.469 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:36.484+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:36.648 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:36.663+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:37.430 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:37.445+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:37.436 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:37.451+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:37.647 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:37.662+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:38.402 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:38.417+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:38.410 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:38.426+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:38.620 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:38.636+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:39.366 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:39.382+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:39.394 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:39.409+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:39.625 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:39.640+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:40.330 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:40.345+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:40.435 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:40.450+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:40.614 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:40.630+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:41.324 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:41.339+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:41.473 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:41.489+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:41.636 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:41.652+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:42.282 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:42.298+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:42.440 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:42.455+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:42.650 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:42.665+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:43.299 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:43.314+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:43.413 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:43.429+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:43.652 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:43.667+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:44.251 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:44.266+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:44.405 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:44.421+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:44.614 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:44.630+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:45.280 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:45.295+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:45.380 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:45.395+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:45.582 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:45.598+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:46.316 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:46.331+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:46.332 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:46.348+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:46.592 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:46.608+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:47.270 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:47.285+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:47.332 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:47.348+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:47.623 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:47.638+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:48.235 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:48.250+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:48.356 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:48.371+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:48.674 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:48.689+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:49.243 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:49.258+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:49.340 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:49.356+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:49.705 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:49.721+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:50.233 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:50.248+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:50.312 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:50.327+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:50.720 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:50.735+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:51.231 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:51.246+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:51.351 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:51.367+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:51.682 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:51.697+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:52.199 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:52.215+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:52.319 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:52.334+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:52.715 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:52.731+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:53.172 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:53.188+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:53.288 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:53.303+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:53.703 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:53.718+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:54.193 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:54.209+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:54.268 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:54.284+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:54.655 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:54.671+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:55.222 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:55.238+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:55.228 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:55.243+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:55.612 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:55.628+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:56.200 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:56.216+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:56.235 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:56.252+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:56.653 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:56.669+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:57.210 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:57.226+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:57.210 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:57.226+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:57.672 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:57.689+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:58.189 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:58.205+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:58.235 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:58.252+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:58.640 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:58.657+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:59.181 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:51:59.198+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:59.238 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:51:59.255+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:51:59.623 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:51:59.639+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:00.161 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:00.177+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:00.254 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:00.271+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:00.653 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:00.669+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:01.163 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:01.179+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:01.271 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:01.287+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:01.631 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:01.647+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:02.194 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:02.210+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:02.299 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:02.315+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:02.628 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:02.642+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:03.235 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:03.252+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:03.283 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:03.299+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:03.626 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T20:52:03.642+0000 7fccf53da700 0 -- 172.21.15.87:0/3193322219 >> [v2:172.21.15.87:6808/114597,v1:172.21.15.87:6809/114597] conn(0x7fccd8033f40 msgr2=0x7fccd803cf00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-24T20:52:03.627 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:03.644+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:04.234 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:04.251+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:04.257 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:04.274+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:04.648 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:04.665+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:05.195 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:05.212+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:05.224 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:05.241+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:05.627 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:05.644+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:06.156 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:06.172+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:06.195 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:06.212+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:06.654 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:06.670+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:07.111 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:07.128+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:07.241 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:07.258+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:07.614 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:07.630+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:08.118 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:08.134+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:08.244 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:08.260+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:08.594 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:08.610+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:09.121 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:09.137+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:09.273 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:09.288+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:09.625 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:09.641+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:10.157 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:10.173+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:10.264 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:10.279+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:10.615 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:10.631+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:11.137 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:11.153+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:11.281 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:11.296+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:11.589 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:11.605+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:12.115 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:12.131+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:12.282 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:12.298+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:12.631 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:12.647+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:13.098 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:13.113+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:13.256 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:13.272+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:13.671 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:13.687+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:14.099 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:14.114+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:14.230 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:14.245+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:14.718 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:14.733+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:15.051 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:15.066+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:15.269 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:15.285+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:15.760 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:15.776+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:16.065 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:16.081+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:16.255 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:16.271+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:16.769 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:16.785+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:17.074 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:17.089+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:17.285 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:17.300+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:17.787 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:17.803+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:18.081 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:18.096+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:18.312 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:18.327+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:18.824 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:18.839+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:19.108 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:19.123+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:19.356 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:19.372+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:19.799 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:19.814+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:20.144 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:20.159+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:20.333 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:20.348+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:20.823 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:20.839+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:21.120 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:21.136+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:21.309 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:21.325+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:21.839 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:21.855+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:22.163 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:22.179+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:22.316 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:22.331+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:22.798 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:22.814+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:23.169 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:23.185+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:23.343 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:23.359+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:23.761 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:23.777+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:24.219 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:24.234+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:24.311 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:24.326+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:24.732 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:24.747+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:25.231 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:25.247+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:25.319 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:25.334+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:25.737 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:25.752+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:26.278 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:26.294+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:26.367 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:26.383+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:26.735 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:26.750+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:27.285 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:27.300+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:27.394 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:27.409+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:27.762 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:27.777+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:28.291 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:28.306+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:28.407 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:28.422+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:28.733 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:28.749+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:29.270 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:29.285+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:29.358 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:29.374+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:29.763 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:29.779+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:30.298 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:30.314+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:30.402 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:30.418+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:30.787 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:30.802+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:31.318 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:31.334+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:31.367 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:31.382+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:31.816 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:31.832+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:32.320 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:32.336+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:32.372 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:32.388+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:32.834 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:32.849+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:33.291 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:33.307+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:33.389 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:33.404+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:33.830 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:33.846+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:34.301 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:34.317+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:34.411 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:34.427+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:34.847 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:34.863+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:35.284 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:35.300+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:35.374 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:35.389+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:35.825 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:35.841+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:36.256 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:36.272+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:36.408 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:36.423+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:36.856 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:36.871+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:37.258 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:37.273+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:37.427 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:37.442+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:37.818 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:37.833+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:38.293 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:38.308+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:38.426 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:38.442+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:38.798 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:38.813+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:39.251 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:39.266+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:39.463 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:39.479+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:39.827 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:39.842+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:40.284 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:40.299+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:40.510 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:40.526+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:40.796 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:40.812+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:41.292 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:41.309+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:41.524 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:41.540+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:41.764 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:41.781+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:42.254 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:42.271+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:42.540 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:42.557+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:42.793 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:42.810+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:43.217 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:43.234+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:43.495 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:43.512+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:43.787 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:43.803+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:44.189 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:44.206+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:44.538 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:44.555+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:44.830 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:44.846+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:45.172 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:45.189+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:45.497 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:45.514+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:45.813 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:45.829+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:46.156 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:46.173+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:46.488 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:46.505+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:46.772 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:46.789+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:47.133 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:47.149+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:47.530 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:47.547+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:47.730 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:47.747+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:48.127 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:48.143+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:48.487 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:48.503+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:48.778 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:48.795+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:49.082 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:49.099+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:49.524 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:49.541+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:49.798 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:49.815+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:50.067 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:50.083+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:50.567 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:50.584+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:50.762 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:50.779+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:51.021 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:51.037+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:51.607 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:51.624+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:51.806 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:51.823+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:52.041 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:52.057+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:52.652 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:52.668+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:52.774 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:52.790+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:53.073 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:53.090+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:53.639 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:53.655+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:53.804 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:53.820+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:54.062 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:54.078+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:54.622 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:54.638+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:54.758 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:54.773+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:55.095 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:55.110+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:55.585 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:55.601+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:55.783 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:55.798+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:56.063 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:56.079+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:56.564 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:56.580+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:56.812 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:56.828+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:57.050 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:57.066+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:57.589 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:57.605+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:57.767 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:57.783+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:58.041 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:58.056+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:58.616 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:58.631+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:58.806 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:58.822+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:59.029 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:52:59.044+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:59.613 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:52:59.628+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:52:59.792 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:52:59.808+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:00.016 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:00.031+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:00.580 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:00.596+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:00.768 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:00.784+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:00.994 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:01.009+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:01.593 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:01.609+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:01.760 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:01.776+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:01.987 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:02.003+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:02.591 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:02.607+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:02.791 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:02.807+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:03.011 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:03.027+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:03.572 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:03.587+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:03.814 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:03.830+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:04.023 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:04.039+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:04.556 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:04.572+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:04.858 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:04.874+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:05.015 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:05.031+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:05.559 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:05.574+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:05.889 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:05.905+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:06.043 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:06.058+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:06.561 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:06.577+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:06.866 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:06.882+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:07.021 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:07.037+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:07.514 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:07.530+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:07.822 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:07.838+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:07.987 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:08.003+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:08.541 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:08.556+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:08.820 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:08.836+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:09.023 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:09.039+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:09.506 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:09.522+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:09.846 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:09.862+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:10.019 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:10.035+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:10.475 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:10.491+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:10.818 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:10.833+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:10.973 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:10.989+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:11.433 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:11.448+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:11.825 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:11.841+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:11.978 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:11.993+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:12.462 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:12.478+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:12.792 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:12.808+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:13.005 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:13.021+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:13.432 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:13.448+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:13.810 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:13.826+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:14.046 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:14.062+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:14.444 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:14.460+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:14.842 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:14.857+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:15.085 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:15.101+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:15.486 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:15.502+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:15.858 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:15.873+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:16.059 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:16.075+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:16.475 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:16.491+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:16.880 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:16.896+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:17.057 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:17.073+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:17.473 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:17.489+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:17.872 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:17.888+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:18.057 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:18.072+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:18.446 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:18.462+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:18.890 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:18.905+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:19.071 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:19.086+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:19.467 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:19.482+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:19.843 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:19.858+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:20.030 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:20.046+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:20.485 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:20.501+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:20.868 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:20.884+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:21.036 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:21.051+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:21.504 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:21.520+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:21.862 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:21.878+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:22.021 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:22.037+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:22.462 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:22.478+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:22.842 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:22.857+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:23.031 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:23.047+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:23.446 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:23.462+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:23.832 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:23.848+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:24.011 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:24.027+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:24.452 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:24.468+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:24.860 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:24.876+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:25.004 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:25.019+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:25.453 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:25.469+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:25.844 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:25.860+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:25.982 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:25.999+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:26.407 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:26.424+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:26.893 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:26.909+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:26.940 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:26.957+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:27.386 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:27.403+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:27.880 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:27.897+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:27.903 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:27.920+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:28.391 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:28.408+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:28.861 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:28.878+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:28.875 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:28.892+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:29.388 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:29.405+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:29.826 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:29.842+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:29.885 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:29.901+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:30.436 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:30.453+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:30.836 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:30.853+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:30.860 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:30.877+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:31.420 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:31.437+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:31.842 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:31.859+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:31.909 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:31.925+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:32.398 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:32.415+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:32.834 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:32.851+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:32.904 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:32.921+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:33.389 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:33.406+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:33.879 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:33.896+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:33.942 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:33.959+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:34.397 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:34.414+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:34.865 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:34.882+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:34.914 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:34.931+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:35.364 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:35.381+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:35.894 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:35.910+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:35.959 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:35.976+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:36.326 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:36.343+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:36.932 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:36.949+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:37.006 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:37.023+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:37.312 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:37.329+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:37.966 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:37.982+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:38.017 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:38.034+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:38.357 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:38.373+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:39.014 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:39.030+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:39.048 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:39.064+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:39.394 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:39.410+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:39.996 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:40.012+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:40.045 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:40.060+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:40.409 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:40.425+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:40.980 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:40.996+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:41.074 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:41.090+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:41.391 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:41.407+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:42.012 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:42.028+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:42.039 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:42.055+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:42.418 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:42.433+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:42.985 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:43.001+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:42.995 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:43.011+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:43.371 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:43.387+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:43.946 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:43.962+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:43.956 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:43.972+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:44.403 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:44.418+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:44.909 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:44.925+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:44.979 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:44.994+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:45.427 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:45.443+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:45.865 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:45.881+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:45.988 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:46.004+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:46.391 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:46.406+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:46.849 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:46.864+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:47.006 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:47.022+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:47.421 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:47.437+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:47.802 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:47.818+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:48.014 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:48.030+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:48.464 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:48.480+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:48.826 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:48.842+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:49.003 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:49.019+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:49.468 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:49.484+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:49.845 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:49.861+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:50.051 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:50.067+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:50.506 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:50.522+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:50.820 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:50.836+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:51.063 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:51.079+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:51.533 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:51.549+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:51.843 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:51.859+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:52.075 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:52.091+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:52.519 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:52.535+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:52.818 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:52.833+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:53.097 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:53.113+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:53.505 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:53.521+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:53.838 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:53.854+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:54.050 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:54.066+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:54.496 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:54.512+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:54.804 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:54.820+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:55.049 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:55.065+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:55.530 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:55.546+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:55.809 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:55.824+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:56.042 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:56.058+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:56.504 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:56.520+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:56.782 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:56.798+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:56.998 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:57.014+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:57.468 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:57.484+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:57.755 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:57.771+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:58.020 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:58.036+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:58.500 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:58.516+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:58.754 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:58.770+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:59.056 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:53:59.072+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:59.526 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:53:59.542+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:53:59.718 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:53:59.734+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:00.063 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:00.079+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:00.540 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:00.556+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:00.712 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:00.727+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:01.075 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:01.091+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:01.590 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:01.606+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:01.670 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:01.686+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:02.114 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:02.130+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:02.579 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:02.595+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:02.627 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:02.643+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:03.116 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:03.132+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:03.603 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:03.619+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:03.608 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:03.624+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:04.078 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:04.094+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:04.564 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:04.580+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:04.609 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:04.625+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:05.073 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:05.089+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:05.594 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:05.610+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:05.596 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:05.612+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:06.099 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:06.115+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:06.575 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:06.591+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:06.617 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:06.632+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:07.070 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:07.085+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:07.592 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:07.607+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:07.603 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:07.619+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:08.088 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:08.104+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:08.638 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:08.654+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:08.642 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:08.658+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:09.113 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:09.129+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:09.609 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:09.626+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:09.678 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:09.695+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:10.072 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:10.088+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:10.572 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:10.589+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:10.672 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:10.689+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:11.106 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:11.114+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:11.558 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:11.575+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:11.646 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:11.663+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:12.056 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:12.073+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:12.587 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:12.604+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:12.606 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:12.623+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:13.042 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:13.059+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:13.602 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:13.619+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:13.609 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:13.626+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:14.068 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:14.084+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:14.584 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:14.601+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:14.649 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:14.666+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:15.055 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:15.072+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:15.634 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:15.650+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:15.662 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:15.680+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:16.072 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:16.089+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:16.624 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:16.641+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:16.695 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:16.712+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:17.116 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:17.133+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:17.599 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:17.616+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:17.697 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:17.714+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:18.075 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:18.092+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:18.634 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:18.651+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:18.726 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:18.743+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:19.123 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:19.140+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:19.674 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:19.691+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:19.738 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:19.755+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:20.105 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:20.122+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:20.696 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:20.713+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:20.739 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:20.756+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:21.083 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:21.099+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:21.670 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:21.687+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:21.766 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:21.783+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:22.067 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:22.084+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:22.681 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:22.695+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:22.801 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:22.819+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:23.100 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:23.116+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:23.636 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:23.652+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:23.818 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:23.833+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:24.066 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:24.082+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:24.648 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:24.664+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:24.849 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:24.865+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:25.091 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:25.107+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:25.661 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:25.677+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:25.839 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:25.855+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:26.129 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:26.145+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:26.633 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:26.648+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:26.889 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:26.905+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:27.085 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:27.101+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:27.607 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:27.624+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:27.928 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:27.944+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:28.135 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:28.151+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:28.572 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:28.588+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:28.923 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:28.939+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:29.153 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:29.169+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:29.539 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:29.555+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:29.883 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:29.899+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:30.119 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:30.135+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:30.523 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:30.540+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:30.852 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:30.868+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:31.105 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:31.122+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:31.548 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:31.564+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:31.809 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:31.825+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:32.059 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:32.075+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:32.510 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:32.526+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:32.854 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:32.870+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:33.066 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:33.082+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:33.556 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:33.572+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:33.827 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:33.843+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:34.116 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:34.132+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:34.558 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:34.574+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:34.841 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:34.857+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:35.116 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:35.132+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:35.520 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:35.536+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:35.875 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:35.891+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:36.109 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:36.125+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:36.534 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:36.550+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:36.892 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:36.908+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:37.100 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:37.115+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:37.555 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:37.571+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:37.927 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:37.943+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:38.137 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:38.153+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:38.526 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:38.542+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:38.971 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:38.987+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:39.153 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:39.169+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:39.564 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:39.580+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:39.994 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:40.010+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:40.197 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:40.213+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:40.524 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:40.540+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:41.031 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:41.048+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:41.197 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:41.213+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:41.521 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:41.537+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:42.054 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:42.070+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:42.364 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:42.198+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:42.548 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:42.564+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:43.034 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:43.050+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:43.208 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:43.224+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:43.549 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:43.565+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:44.037 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:44.053+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:44.172 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:44.188+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:44.553 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:44.569+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:45.078 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:45.094+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:45.141 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:45.157+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:45.509 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:45.524+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:46.101 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:46.117+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:46.117 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:46.134+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:46.511 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:46.527+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:47.053 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:47.069+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:47.111 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:47.126+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:47.469 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:47.485+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:48.025 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:48.041+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:48.097 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:48.113+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:48.486 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:48.502+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:49.037 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:49.053+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:49.132 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:49.148+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:49.528 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:49.544+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:50.024 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:50.041+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:50.167 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:50.183+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:50.533 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:50.549+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:51.033 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:51.049+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:51.122 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:51.138+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:51.499 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:51.515+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:52.014 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:52.030+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:52.074 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:52.090+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:52.505 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:52.521+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:52.966 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:52.983+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:53.114 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:53.131+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:53.484 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:53.500+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:53.932 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:53.948+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:54.138 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:54.155+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:54.470 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:54.487+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:54.892 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:54.908+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:55.164 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:55.180+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:55.455 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:55.473+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:55.866 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:55.883+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:56.129 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:56.146+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:56.423 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:56.440+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:56.913 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:56.930+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:57.085 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:57.102+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:57.388 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:57.403+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:57.884 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:57.901+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:58.131 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:58.148+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:58.404 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:58.421+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:58.880 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:58.897+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:59.084 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:54:59.101+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:59.439 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:54:59.456+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:54:59.870 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:54:59.887+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:00.064 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:00.081+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:00.423 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:00.440+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:00.827 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:00.844+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:01.048 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:01.065+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:01.380 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:01.397+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:01.803 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:01.820+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:02.046 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:02.063+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:02.347 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:02.364+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:02.775 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:02.791+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:03.033 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:03.050+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:03.364 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:03.381+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:03.764 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:03.781+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:04.014 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:04.031+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:04.392 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:04.409+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:04.794 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:04.811+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:05.049 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:05.066+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:05.405 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:05.422+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:05.767 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:05.784+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:06.098 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:06.115+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:06.375 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:06.392+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:06.796 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:06.813+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:07.113 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:07.130+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:07.404 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:07.421+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:07.818 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:07.835+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:08.088 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:08.105+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:08.389 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:08.407+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:08.784 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:08.800+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:09.046 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:09.062+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:09.408 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:09.424+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:09.806 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:09.822+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:10.032 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:10.048+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:10.450 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:10.466+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:10.808 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:10.824+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:11.033 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:11.049+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:11.425 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:11.441+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:11.782 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:11.798+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:12.056 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:12.072+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:12.424 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:12.440+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:12.786 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:12.802+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:13.036 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:13.052+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:13.388 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:13.404+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:13.770 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:13.786+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:14.010 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:14.026+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:14.383 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:14.399+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:14.742 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:14.759+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:14.998 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:15.014+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:15.372 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:15.388+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:15.748 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:15.764+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:15.957 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:15.973+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:16.379 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:16.395+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:16.797 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:16.813+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:16.979 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:16.995+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:17.340 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:17.356+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:17.812 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:17.828+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:17.940 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:17.956+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:18.371 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:18.387+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:18.860 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:18.876+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:18.921 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:18.937+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:19.337 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:19.353+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:19.843 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:19.860+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:19.878 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:19.894+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:20.346 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:20.362+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:20.864 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:20.880+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:20.865 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:20.882+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:21.345 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:21.362+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:21.850 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:21.866+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:21.856 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:21.872+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:22.348 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:22.364+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:22.804 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:22.820+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:22.829 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:22.846+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:23.396 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:23.412+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:23.804 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:23.820+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:23.814 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:23.831+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:24.365 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:24.381+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:24.784 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:24.800+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:24.786 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:24.802+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:25.373 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:25.389+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:25.736 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:25.752+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:25.760 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:25.777+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:26.383 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:26.399+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:26.693 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:26.709+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:26.719 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:26.735+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:27.360 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:27.377+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:27.647 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:27.664+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:27.727 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:27.743+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:28.339 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:28.355+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:28.650 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:28.666+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:28.698 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:28.715+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:29.332 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:29.348+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:29.690 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:29.706+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:29.715 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:29.731+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:30.287 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:30.304+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:30.683 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:30.699+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:30.738 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:30.754+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:31.265 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:31.281+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:31.719 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:31.735+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:31.784 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:31.801+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:32.299 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:32.316+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:32.742 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:32.758+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:32.787 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:32.803+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:33.269 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:33.286+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:33.695 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:33.711+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:33.744 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:33.760+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:34.270 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:34.286+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:34.714 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:34.730+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:34.768 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:34.785+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:35.290 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:35.306+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:35.705 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:35.721+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:35.725 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:35.741+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:36.258 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:36.275+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:36.746 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:36.762+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:36.753 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:36.770+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:37.247 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:37.264+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:37.774 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:37.790+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:37.783 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:37.799+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:38.223 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:38.239+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:38.756 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:38.772+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:38.819 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:38.835+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:39.185 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:39.202+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:39.795 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:39.811+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:39.811 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:39.827+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:40.169 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:40.186+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:40.775 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:40.793+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:40.811 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:40.828+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:41.166 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:41.184+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:41.732 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:41.750+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:41.789 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:41.807+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:42.177 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:42.194+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:42.742 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:42.759+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:42.780 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:42.798+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:43.162 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:43.179+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:43.723 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:43.740+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:43.812 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:43.829+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:44.143 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:44.160+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:44.764 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:44.781+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:44.822 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:44.839+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:45.191 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:45.208+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:45.793 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:45.810+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:45.816 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:45.833+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:46.229 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:46.247+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:46.811 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:46.828+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:46.841 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:46.858+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:47.192 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:47.210+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:47.777 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:47.794+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:47.839 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:47.857+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:48.232 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:48.249+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:48.753 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:48.771+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:48.837 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:48.855+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:49.187 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:49.204+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:49.726 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:49.743+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:49.797 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:49.815+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:50.175 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:50.193+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:50.760 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:50.777+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:50.763 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:50.780+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:51.193 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:51.211+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:51.729 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:51.747+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:51.776 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:51.793+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:52.147 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:52.164+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:52.683 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:52.700+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:52.729 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:52.747+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:53.174 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:53.191+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:53.664 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:53.680+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:53.757 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:53.773+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:54.131 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:54.147+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:54.706 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:54.723+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:54.748 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:54.765+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:55.121 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:55.137+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:55.671 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:55.687+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:55.766 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:55.782+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:56.092 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:56.108+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:56.637 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:56.653+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:56.729 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:56.746+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:57.060 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:57.076+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:57.677 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:57.693+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:57.734 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:57.750+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:58.047 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:58.063+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:58.670 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:58.686+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:58.743 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:58.760+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:59.087 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:55:59.104+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:59.678 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:55:59.694+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:55:59.791 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:55:59.808+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:00.099 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:00.115+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:00.655 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:00.672+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:00.801 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:00.818+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:01.099 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:01.115+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:01.647 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:01.664+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:01.804 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:01.821+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:02.070 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:02.087+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:02.607 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:02.623+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:02.828 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:02.845+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:03.112 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:03.129+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:03.645 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:03.662+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:03.835 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:03.851+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:04.098 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:04.114+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:04.670 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:04.687+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:04.826 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:04.842+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:05.113 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:05.129+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:05.682 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:05.698+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:05.815 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:05.831+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:06.161 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:06.178+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:06.666 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:06.683+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:06.833 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:06.849+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:07.131 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:07.148+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:07.702 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:07.718+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:07.824 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:07.840+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:08.124 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:08.140+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:08.724 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:08.740+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:08.841 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:08.857+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:09.126 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:09.143+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:09.728 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:09.744+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:09.834 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:09.850+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:10.113 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:10.130+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:10.733 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:10.749+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:10.871 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:10.888+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:11.111 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:11.128+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:11.708 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:11.725+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:11.841 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:11.858+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:12.135 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:12.151+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:12.690 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:12.706+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:12.886 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:12.903+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:13.099 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:13.115+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:13.732 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:13.748+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:13.842 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:13.859+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:14.123 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:14.140+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:14.760 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:14.776+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:14.874 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:14.890+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:15.170 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:15.187+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:15.778 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:15.794+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:15.902 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:15.918+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:16.167 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:16.184+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:16.788 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:16.804+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:16.934 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:16.950+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:17.206 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:17.223+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:17.819 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:17.835+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:17.947 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:17.963+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:18.226 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:18.243+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:18.825 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:18.841+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:18.974 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:18.990+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:19.223 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:19.239+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:19.832 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:19.848+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:19.990 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:20.006+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:20.213 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:20.230+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:20.880 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:20.896+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:20.994 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:21.010+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:21.239 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:21.256+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:21.874 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:21.890+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:22.023 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:22.040+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:22.270 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:22.286+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:22.855 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:22.871+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:22.997 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:23.011+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:23.278 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:23.295+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:23.842 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:23.859+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:23.976 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:23.993+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:24.291 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:24.307+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:24.871 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:24.888+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:25.004 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:25.020+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:25.253 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:25.271+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:25.882 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:25.899+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:26.009 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:26.026+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:26.282 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:26.299+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:26.872 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:26.889+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:26.961 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:26.979+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:27.329 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:27.346+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:27.832 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:27.850+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:27.995 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:28.012+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:28.303 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:28.321+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:28.790 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:28.807+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:28.972 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:28.989+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:29.336 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:29.353+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:29.763 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:29.780+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:29.942 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:29.959+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:30.381 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:30.399+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:30.763 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:30.780+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:30.935 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:30.952+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:31.422 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:31.439+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:31.768 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:31.785+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:31.894 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:31.911+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:32.435 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:32.452+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:32.744 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:32.762+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:32.868 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:32.885+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:33.406 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:33.423+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:33.741 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:33.759+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:33.859 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:33.877+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:34.433 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:34.450+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:34.759 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:34.777+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:34.885 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:34.902+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:35.426 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:35.443+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:35.801 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:35.818+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:35.931 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:35.949+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:36.433 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:36.450+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:36.771 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:36.789+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:36.912 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:36.930+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:37.413 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:37.431+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:37.758 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:37.775+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:37.952 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:37.969+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:38.389 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:38.406+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:38.743 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:38.759+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:38.962 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:38.978+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:39.404 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:39.420+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:39.790 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:39.807+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:39.991 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:40.007+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:40.446 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:40.463+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:40.803 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:40.819+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:41.037 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:41.054+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:41.484 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:41.500+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:41.799 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:41.815+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:41.993 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:42.009+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:42.498 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:42.515+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:42.812 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:42.829+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:42.959 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:42.976+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:43.489 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:43.504+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:43.856 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:43.872+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:43.942 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:43.959+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:44.517 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:44.533+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:44.839 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:44.856+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:44.907 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:44.924+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:45.479 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:45.496+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:45.849 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:45.866+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:45.913 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:45.929+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:46.490 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:46.507+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:46.845 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:46.862+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:46.948 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:46.965+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:47.477 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:47.494+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:47.801 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:47.817+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:47.998 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:48.014+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:48.504 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:48.521+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:48.823 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:48.839+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:48.953 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:48.969+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:49.528 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:49.545+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:49.791 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:49.808+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:49.947 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:49.964+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:50.496 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:50.512+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:50.799 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:50.816+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:50.947 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:50.964+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:51.523 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:51.539+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:51.788 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:51.805+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:51.929 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:51.945+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:52.487 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:52.503+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:52.779 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:52.796+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:52.975 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:52.992+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:53.537 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:53.553+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:53.733 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:53.750+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:53.977 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:53.994+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:54.545 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:54.561+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:54.780 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:54.797+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:54.953 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:54.970+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:55.555 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:55.572+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 26 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:55.796 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:55.812+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:55.967 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:55.984+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:56.583 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:56.600+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:56.807 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:56.824+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:56.951 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:56.968+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:57.576 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:57.592+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:57.793 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:57.810+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:57.945 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:57.962+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:58.542 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:58.558+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:58.760 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:58.777+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:58.948 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:58.965+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:59.568 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:56:59.585+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:59.791 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:56:59.808+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:56:59.934 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:56:59.951+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:00.596 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:00.613+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:00.760 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:00.777+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:00.972 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:00.988+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:01.568 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:01.585+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:01.717 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:01.734+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:01.941 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:01.958+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:02.550 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:02.566+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:02.738 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:02.755+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:02.983 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:03.000+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:03.592 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:03.609+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:03.745 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:03.761+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:03.997 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:04.014+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:04.640 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:04.657+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:04.699 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:04.716+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:05.031 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:05.048+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:05.645 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:05.662+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:05.653 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:05.670+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:06.073 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:06.090+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:06.610 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:06.626+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:06.634 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:06.651+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:07.062 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:07.079+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:07.655 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:07.671+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:07.658 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:07.675+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:08.068 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:08.084+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:08.649 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:08.666+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:08.661 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:08.678+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:09.073 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:09.089+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:09.650 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:09.666+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:09.660 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:09.677+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:10.099 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:10.117+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:10.650 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:10.667+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:10.654 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:10.672+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:11.112 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:11.130+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:11.613 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:11.631+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:11.673 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:11.691+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:12.131 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:12.149+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:12.662 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:12.680+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:12.713 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:12.731+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:13.083 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:13.100+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:13.700 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:13.718+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:13.731 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:13.749+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:14.128 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:14.146+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:14.685 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:14.702+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:14.724 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:14.742+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:15.175 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:15.193+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:15.716 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:15.734+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:15.758 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:15.775+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:16.162 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:16.180+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:16.727 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:16.745+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:16.742 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:16.760+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:17.180 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:17.198+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:17.749 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:17.767+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:17.768 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:17.786+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:18.201 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:18.219+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:18.746 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:18.764+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:18.756 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:18.774+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:19.166 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:19.184+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:19.711 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:19.729+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:19.760 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:19.778+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:20.171 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:20.189+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:20.733 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:20.751+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:20.810 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:20.828+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:21.174 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:21.192+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:21.689 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:21.707+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:21.850 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:21.868+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:22.198 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:22.215+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:22.723 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:22.740+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:22.852 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:22.870+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:23.230 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:23.247+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:23.731 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:23.748+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:23.851 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:23.868+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:24.218 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:24.235+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:24.686 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:24.703+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:24.864 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:24.881+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:25.186 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:25.203+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:25.647 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:25.663+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:25.864 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:25.880+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:26.138 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:26.155+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:26.630 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:26.646+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:26.868 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:26.885+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:27.097 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:27.114+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:27.674 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:27.691+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:27.885 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:27.901+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:28.055 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:28.072+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:28.634 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:28.651+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:28.899 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:28.915+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:29.102 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:29.119+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:29.626 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:29.643+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:29.858 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:29.875+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:30.064 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:30.081+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:30.651 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:30.668+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:30.908 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:30.925+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:31.080 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:31.097+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:31.675 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:31.692+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:31.865 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:31.881+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:32.128 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:32.145+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:32.704 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:32.721+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:32.893 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:32.910+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:33.148 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:33.165+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:33.659 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:33.676+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:33.893 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:33.910+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:34.152 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:34.168+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:34.695 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:34.712+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:34.868 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:34.885+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:35.131 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:35.147+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:35.699 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:35.716+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:35.883 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:35.900+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:36.087 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:36.104+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:36.727 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:36.744+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:36.900 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:36.917+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:37.078 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:37.095+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:37.700 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:37.717+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:37.940 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:37.957+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:38.062 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:38.079+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:38.687 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:38.703+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:38.929 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:38.946+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:39.100 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:39.117+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:39.639 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:39.655+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:39.890 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:39.907+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:40.107 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:40.124+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:40.636 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:40.653+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:40.894 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:40.910+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:41.141 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:41.158+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:41.660 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:41.676+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:41.911 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:41.928+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:42.148 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:42.165+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:42.686 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:42.703+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:42.936 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:42.953+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:43.149 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:43.165+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:43.721 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:43.737+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:43.888 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:43.904+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:44.168 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:44.185+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:44.751 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:44.768+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:44.848 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:44.864+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:45.138 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:45.155+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:45.706 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:45.723+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:45.876 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:45.892+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:46.114 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:46.130+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:46.659 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:46.676+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:46.909 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:46.926+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:47.112 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:47.128+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:47.697 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:47.713+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:47.950 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:47.967+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:48.139 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:48.156+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:48.650 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:48.667+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:48.986 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:49.003+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:49.120 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:49.137+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:49.622 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:49.639+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:49.975 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:49.992+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:50.146 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:50.163+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:50.591 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:50.607+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:50.942 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:50.959+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:51.125 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:51.142+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:51.599 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:51.616+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:51.930 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:51.947+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:52.095 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:52.112+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:52.641 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:52.657+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:52.895 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:52.912+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:53.071 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:53.088+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:53.683 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:53.700+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:53.925 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:53.942+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:54.028 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:54.045+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:54.649 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:54.666+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:54.920 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:54.938+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:55.065 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:55.083+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:55.645 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:55.663+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:55.943 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:55.961+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:56.102 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:56.120+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:56.616 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:56.633+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:56.922 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:56.940+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:57.079 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:57.097+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:57.630 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:57.647+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:57.957 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:57.975+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:58.112 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:58.130+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:58.666 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:58.684+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:58.976 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:57:58.994+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:59.068 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:57:59.086+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:57:59.649 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:57:59.667+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:00.000 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:00.017+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:00.040 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:00.058+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:00.620 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:00.637+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:00.987 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:01.005+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:01.057 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:01.075+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:01.585 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:01.603+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:01.947 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:01.965+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:02.103 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:02.121+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:02.540 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:02.558+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:02.904 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:02.922+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:03.089 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:03.107+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:03.524 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:03.542+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:03.932 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:03.949+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:04.104 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:04.121+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:04.477 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:04.495+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:04.956 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:04.973+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:05.075 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:05.093+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:05.493 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:05.511+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:05.994 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:06.012+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:06.096 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:06.114+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:06.460 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:06.478+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:06.968 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:06.986+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:07.144 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:07.162+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:07.470 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:07.488+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:07.953 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:07.969+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:08.170 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:08.187+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:08.421 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:08.438+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:09.002 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:09.019+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:09.214 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:09.231+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:09.464 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:09.481+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:10.046 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:10.063+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:10.179 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:10.196+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:10.458 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:10.475+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:11.047 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:11.064+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:11.212 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:11.229+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:11.472 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:11.489+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:12.058 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:12.074+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:12.179 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:12.196+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:12.481 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:12.498+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:13.092 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:13.109+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:13.139 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:13.156+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:13.454 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:13.471+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:14.074 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:14.091+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:14.148 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:14.164+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:14.418 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:14.435+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:15.084 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:15.101+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:15.183 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:15.200+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:15.404 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:15.421+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:16.079 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:16.095+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:16.140 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:16.157+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:16.378 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:16.395+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:17.070 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:17.087+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:17.183 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:17.200+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:17.340 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:17.357+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:18.109 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:18.126+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:18.168 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:18.185+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:18.382 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:18.399+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:19.136 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:19.152+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:19.168 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:19.185+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:19.337 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:19.354+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:20.087 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:20.103+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:20.218 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:20.235+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:20.289 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:20.306+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:21.049 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:21.066+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:21.208 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:21.225+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:21.256 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:21.273+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:22.075 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:22.091+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:22.235 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:22.251+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:22.238 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:22.256+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:23.085 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:23.102+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:23.205 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:23.221+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:23.210 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:23.227+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:24.082 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:24.099+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:24.205 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:24.222+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:24.248 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:24.265+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:25.042 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:25.059+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:25.199 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:25.216+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:25.298 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:25.315+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:26.092 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:26.109+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:26.229 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:26.246+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:26.275 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:26.292+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:27.130 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:27.147+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:27.227 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:27.244+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:27.289 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:27.305+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:28.150 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:28.167+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:28.240 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:28.257+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:28.319 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:28.336+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:29.104 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:29.121+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:29.229 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:29.246+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:29.271 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:29.288+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:30.112 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:30.129+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:30.244 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:30.261+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:30.250 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:30.267+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:31.136 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:31.153+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:31.212 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:31.229+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:31.247 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:31.264+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:32.106 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:32.123+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:32.177 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:32.194+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:32.276 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:32.293+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:33.146 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:33.163+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:33.211 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:33.227+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:33.262 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:33.279+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:34.157 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:34.173+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:34.162 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:34.180+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:34.232 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:34.249+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:35.124 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:35.141+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:35.144 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:35.161+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:35.189 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:35.206+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:36.098 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:36.115+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:36.106 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:36.123+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:36.211 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:36.228+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:37.067 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:37.084+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:37.090 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:37.107+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:37.233 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:37.250+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:38.046 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:38.063+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:38.119 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:38.136+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:38.254 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:38.271+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:39.055 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:39.072+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:39.084 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:39.101+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:39.287 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:39.304+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:40.019 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:40.037+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:40.097 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:40.115+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:40.278 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:40.296+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:40.314 INFO:tasks.workunit.client.0.smithi087.stderr:2023-12-24T20:58:40.332+0000 7fcce17fa700 -1 received signal: Terminated from timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/cli_generic.sh (PID: 120946) UID: 1000 2023-12-24T20:58:40.314 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-24T20:58:40.983 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:41.001+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:41.058 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:41.076+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:41.248 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:41.266+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:42.021 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:42.038+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:42.021 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:42.039+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:42.207 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:42.225+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:43.043 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:43.061+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:43.068 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:43.085+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:43.230 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:43.248+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:44.052 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:44.070+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:44.077 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:44.095+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:44.251 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:44.269+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:45.090 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:45.108+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:45.113 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:45.131+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:45.261 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:45.279+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:46.093 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:46.111+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:46.123 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:46.141+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:46.288 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:46.306+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:47.091 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:47.109+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:47.116 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:47.134+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:47.240 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:47.258+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:48.110 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:48.128+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:48.122 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:48.140+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:48.288 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:48.306+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:49.097 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:49.114+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:49.137 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:49.155+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:49.321 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:49.339+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:50.104 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:50.122+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:50.175 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:50.193+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:50.349 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:50.368+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:51.141 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:51.159+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:51.194 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:51.212+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:51.365 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:51.383+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:52.162 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:52.180+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:52.177 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:52.196+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:52.389 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:52.407+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:53.131 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:53.148+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:53.186 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:53.203+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:53.422 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:53.439+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:54.089 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:54.106+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:54.192 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:54.209+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:54.383 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:54.400+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:55.082 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:55.099+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:55.161 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:55.178+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:55.347 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:55.364+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:56.076 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:56.093+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:56.117 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:56.134+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:56.313 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:56.330+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:57.117 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:57.133+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:57.142 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:57.159+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:57.286 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:57.303+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:58.099 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:58.116+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:58.167 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:58.183+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:58.308 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:58.325+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:59.059 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:58:59.076+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:59.132 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:58:59.149+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:58:59.354 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:58:59.371+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:00.096 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:00.113+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:00.110 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:00.128+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:00.364 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:00.380+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:01.047 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:01.064+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:01.088 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:01.105+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:01.361 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:01.378+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:01.999 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:02.016+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:02.096 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:02.113+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:02.335 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:02.352+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:03.008 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:03.025+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:03.081 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:03.098+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:03.319 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:03.336+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:03.987 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:04.004+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:04.058 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:04.075+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:04.347 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:04.364+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:05.001 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:05.018+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:05.106 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:05.123+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:05.360 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:05.377+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:06.032 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:06.049+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:06.132 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:06.149+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:06.391 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:06.408+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:07.025 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:07.042+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:07.136 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:07.153+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:07.418 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:07.435+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:07.986 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:08.003+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:08.102 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:08.119+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:08.423 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:08.440+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:09.010 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:09.027+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:09.150 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:09.167+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:09.426 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:09.442+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:09.985 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:10.002+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:10.175 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:10.193+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:10.423 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:10.441+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:11.026 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:11.043+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:11.168 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:11.185+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:11.448 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:11.465+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:12.017 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:12.034+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:12.217 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:12.234+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:12.405 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:12.423+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:12.986 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:13.003+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:13.209 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:13.226+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:13.414 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:13.431+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:14.027 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:14.044+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:14.196 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:14.213+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:14.449 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:14.466+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:15.076 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:15.093+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:15.189 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:15.206+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:15.464 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:15.481+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:16.056 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:16.073+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:16.174 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:16.191+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:16.513 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:16.530+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:17.070 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:17.088+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:17.179 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:17.196+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:17.517 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:17.534+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:18.052 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:18.069+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:18.224 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:18.241+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:18.485 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:18.502+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:19.009 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:19.027+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:19.182 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:19.199+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:19.519 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:19.536+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:20.020 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:20.037+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:20.213 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:20.230+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:20.508 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:20.525+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:20.983 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:21.000+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:21.187 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:21.205+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:21.515 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:21.533+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:21.977 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:21.994+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:22.140 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:22.157+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:22.539 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:22.556+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:22.932 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:22.950+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:23.140 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:23.157+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:23.578 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:23.595+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:23.944 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:23.961+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:24.180 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:24.197+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:24.568 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:24.586+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:24.900 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:24.918+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:25.218 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:25.236+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:25.528 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:25.546+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:25.859 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:25.877+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:26.188 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:26.206+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:26.495 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:26.514+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:26.901 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:26.918+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:27.223 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:27.241+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:27.488 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:27.506+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:27.911 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:27.929+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:28.236 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:28.254+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:28.489 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:28.507+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:28.862 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:28.880+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:29.276 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:29.294+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:29.476 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:29.494+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:29.892 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:29.910+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:30.289 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:30.307+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:30.502 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:30.520+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:30.882 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:30.901+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:31.257 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:31.275+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:31.459 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:31.478+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:31.899 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:31.917+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:32.279 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:32.297+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:32.431 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:32.450+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:32.929 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:32.947+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:33.262 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:33.280+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:33.397 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:33.415+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:33.879 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:33.897+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:34.227 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:34.245+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:34.349 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:34.367+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:34.869 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:34.887+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:35.193 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:35.211+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:35.358 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:35.377+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:35.852 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:35.871+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:36.184 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:36.203+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:36.391 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:36.410+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:36.812 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:36.831+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:37.138 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:37.156+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:37.370 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:37.389+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:37.829 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:37.846+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:38.154 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:38.171+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:38.381 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:38.398+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:38.831 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:38.848+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:39.112 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:39.129+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:39.362 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:39.379+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:39.811 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:39.828+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:40.100 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:40.117+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:40.315 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: 2023-12-24T20:59:40.321 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:40.339+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:40.776 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:40.793+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:41.139 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:41.156+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:41.328 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:41.345+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:41.789 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:41.806+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:42.187 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:42.204+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:42.358 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:42.375+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:42.832 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:42.849+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:43.159 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:43.176+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:43.397 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:43.415+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:43.810 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:43.827+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:44.120 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:44.137+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:44.360 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:44.377+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:44.763 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:44.780+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:45.103 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:45.120+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:45.350 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:45.368+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:45.796 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:45.813+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:46.084 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:46.101+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:46.345 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:46.363+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:46.779 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:46.796+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:47.128 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:47.146+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:47.323 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:47.340+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:47.815 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:47.832+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:48.106 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:48.123+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:48.305 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:48.322+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:48.833 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:48.850+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:49.083 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:49.100+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:49.285 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:49.302+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:49.871 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:49.888+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:50.120 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:50.137+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:50.274 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:50.291+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:50.836 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:50.853+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:51.143 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:51.160+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:51.317 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:51.335+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:51.793 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:51.810+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:52.182 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:52.199+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:52.316 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:52.333+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:52.786 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:52.803+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:53.159 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:53.176+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:53.348 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:53.365+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:53.805 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:53.822+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:54.200 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:54.217+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:54.330 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:54.347+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:54.837 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:54.854+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:55.170 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:55.188+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:55.354 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:55.370+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:55.887 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:55.904+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:56.170 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:56.187+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:56.319 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:56.337+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:56.845 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:56.862+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:57.144 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:57.161+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:57.323 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:57.340+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:57.828 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:57.846+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:58.118 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:58.136+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:58.373 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:58.391+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:58.823 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:58.840+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:59.163 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T20:59:59.180+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:59.370 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T20:59:59.387+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T20:59:59.824 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T20:59:59.842+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:00.165 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:00.182+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:00.338 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:00.355+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:00.781 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:00.798+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:01.215 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:01.232+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:01.355 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:01.372+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:01.736 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:01.753+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:02.196 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:02.214+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:02.385 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:02.402+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:02.756 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:02.773+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:03.238 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:03.255+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:03.418 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:03.435+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:03.743 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:03.760+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:04.223 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:04.241+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:04.461 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:04.478+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:04.770 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:04.787+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:05.178 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:05.195+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:05.486 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:05.504+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:05.727 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:05.744+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:06.183 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:06.200+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:06.440 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:06.457+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:06.702 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:06.720+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:07.223 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:07.241+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:07.475 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:07.492+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:07.711 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:07.729+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:08.214 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:08.231+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:08.429 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:08.447+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:08.662 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:08.679+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:09.233 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:09.251+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:09.381 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:09.400+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:09.620 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:09.638+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:10.257 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:10.275+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:10.336 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:10.354+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:10.624 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:10.642+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:11.232 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:11.250+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:11.306 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:11.324+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:11.610 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:11.628+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:12.269 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:12.287+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:12.335 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:12.353+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:12.631 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:12.649+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:13.253 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:13.271+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:13.296 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:13.314+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:13.648 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:13.666+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:14.212 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:14.230+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:14.273 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:14.291+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:14.698 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:14.716+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:15.259 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:15.277+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:15.273 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:15.291+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:15.649 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:15.667+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:16.226 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:16.244+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:16.241 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:16.259+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:16.660 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:16.677+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:17.193 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:17.211+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:17.214 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:17.232+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:17.666 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:17.683+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:18.190 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:18.208+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:18.210 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:18.228+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:18.674 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:18.691+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:19.176 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:19.194+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:19.205 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:19.222+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:19.665 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:19.683+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:20.172 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:20.189+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:20.196 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:20.214+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:20.686 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:20.704+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:21.154 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:21.172+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:21.174 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:21.192+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:21.667 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:21.684+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:22.105 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:22.123+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:22.126 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:22.143+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:22.713 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:22.729+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:23.083 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:23.100+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:23.150 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:23.167+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:23.738 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:23.754+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:24.102 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:24.119+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:24.111 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:24.128+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:24.733 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:24.749+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:25.064 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:25.080+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:25.130 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:25.147+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:25.767 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:25.783+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:26.069 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:26.086+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:26.118 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:26.134+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:26.779 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:26.796+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:27.061 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:27.077+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:27.146 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:27.163+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:27.731 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:27.747+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:28.067 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:28.083+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:28.186 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:28.203+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:28.757 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:28.774+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:29.021 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:29.037+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:29.142 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:29.159+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:29.759 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:29.776+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:30.061 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:30.077+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:30.118 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:30.135+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:30.793 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:30.809+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:31.080 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:31.096+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:31.082 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:31.098+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:31.744 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:31.760+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:32.032 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:32.048+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:32.054 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:32.071+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:32.718 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:32.735+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:33.006 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:33.023+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:33.092 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:33.108+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:33.724 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:33.740+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:34.023 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:34.040+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:34.141 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:34.157+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:34.763 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:34.779+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:34.988 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:35.004+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:35.157 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:35.173+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:35.774 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:35.790+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:35.949 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:35.965+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:36.157 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:36.173+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:36.810 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:36.826+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:36.993 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:37.009+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:37.137 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:37.153+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:37.763 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:37.779+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:37.996 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:38.012+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:38.132 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:38.148+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:38.740 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:38.756+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:39.004 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:39.020+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:39.129 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:39.145+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:39.732 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:39.748+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:40.013 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:40.029+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:40.171 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:40.187+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:40.316 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: 2023-12-24T21:00:40.316 INFO:tasks.workunit:Stopping ['rbd/cli_generic.sh'] on client.0... 2023-12-24T21:00:40.316 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-12-24T21:00:40.584 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/cli_generic.sh) on smithi087 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/cli_generic.sh' 2023-12-24T21:00:40.654 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=da470ffc492d45f08538d0344fcdd745 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/cli_generic.sh) on smithi087 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/cli_generic.sh' 2023-12-24T21:00:40.656 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-24T21:00:40.665 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-12-24T21:00:40.665 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-24T21:00:40.747 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:40.763+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:40.976 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:40.992+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:41.000 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:00:41.000 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":162,"fsid":"0f2278f5-9601-4df5-8109-682fbac125ff","created":"2023-12-24T17:57:27.057964+0000","modified":"2023-12-24T19:00:52.539743+0000","last_up_change":"2023-12-24T17:57:36.716834+0000","last_in_change":"2023-12-24T17:57:28.019393+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-24T17:57:36.963424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-24T17:57:42.305786+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"159","last_force_op_resend":"18","last_force_op_resend_prenautilus":"18","last_force_op_resend_preluminous":"18","auid":0,"snap_mode":"selfmanaged","snap_seq":93,"snap_epoch":159,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[3],"tier_of":-1,"read_tier":3,"write_tier":3,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cache","create_time":"2023-12-24T17:57:49.828478+0000","flags":8201,"flags_names":"hashpspool,incomplete_clones,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"159","last_force_op_resend":"18","last_force_op_resend_prenautilus":"18","last_force_op_resend_preluminous":"18","auid":0,"snap_mode":"selfmanaged","snap_seq":93,"snap_epoch":159,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":2,"read_tier":-1,"write_tier":-1,"cache_mode":"writeback","target_max_bytes":0,"target_max_objects":250,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"bloom","false_positive_probability":0.050000000000000003,"target_size":0,"seed":0},"hit_set_period":60,"hit_set_count":8,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[1000000,1000000,1000000,1000000,1000000,1000000,1000000,1000000],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2c99abb1-7a79-494e-a998-e2942573c6ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":114598},{"type":"v1","addr":"172.21.15.87:6817","nonce":114598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":114598},{"type":"v1","addr":"172.21.15.87:6819","nonce":114598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":114598},{"type":"v1","addr":"172.21.15.87:6823","nonce":114598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":114598},{"type":"v1","addr":"172.21.15.87:6821","nonce":114598}]},"public_addr":"172.21.15.87:6817/114598","cluster_addr":"172.21.15.87:6819/114598","heartbeat_back_addr":"172.21.15.87:6823/114598","heartbeat_front_addr":"172.21.15.87:6821/114598","state":["exists","up"]},{"osd":1,"uuid":"2f9896f4-15d2-40c9-8bd7-05fe1e6d8b61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":114597},{"type":"v1","addr":"172.21.15.87:6809","nonce":114597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":114597},{"type":"v1","addr":"172.21.15.87:6811","nonce":114597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":114597},{"type":"v1","addr":"172.21.15.87:6815","nonce":114597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":114597},{"type":"v1","addr":"172.21.15.87:6813","nonce":114597}]},"public_addr":"172.21.15.87:6809/114597","cluster_addr":"172.21.15.87:6811/114597","heartbeat_back_addr":"172.21.15.87:6815/114597","heartbeat_front_addr":"172.21.15.87:6813/114597","state":["exists","up"]},{"osd":2,"uuid":"0403cd42-78bb-43a2-9809-62b634e0e5df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":114582},{"type":"v1","addr":"172.21.15.87:6801","nonce":114582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":114582},{"type":"v1","addr":"172.21.15.87:6803","nonce":114582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":114582},{"type":"v1","addr":"172.21.15.87:6807","nonce":114582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":114582},{"type":"v1","addr":"172.21.15.87:6805","nonce":114582}]},"public_addr":"172.21.15.87:6801/114582","cluster_addr":"172.21.15.87:6803/114582","heartbeat_back_addr":"172.21.15.87:6807/114582","heartbeat_front_addr":"172.21.15.87:6805/114582","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-24T17:57:33.148164+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-24T17:57:33.220152+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-24T17:57:33.135613+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":4,"length":1}]},{"pool":5,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-24T21:00:41.014 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-24T21:00:41.015 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:00:41.168 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:41.184+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:41.375 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:00:41.377 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:00:41.388 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":5649,"stamp":"2023-12-24T21:00:40.578839+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,2,2,6,12,516],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001844"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2824,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544349+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T19:00:22.544349+0000","last_peered":"2023-12-24T19:00:22.544349+0000","last_clean":"2023-12-24T19:00:22.544349+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T19:00:22.544349+0000","last_undegraded":"2023-12-24T19:00:22.544349+0000","last_fullsized":"2023-12-24T19:00:22.544349+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T20:39:55.726631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3618,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544409+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T19:00:22.544409+0000","last_peered":"2023-12-24T19:00:22.544409+0000","last_clean":"2023-12-24T19:00:22.544409+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T19:00:22.544409+0000","last_undegraded":"2023-12-24T19:00:22.544409+0000","last_fullsized":"2023-12-24T19:00:22.544409+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T01:55:17.265710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2981,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544459+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T19:00:22.544459+0000","last_peered":"2023-12-24T19:00:22.544459+0000","last_clean":"2023-12-24T19:00:22.544459+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T19:00:22.544459+0000","last_undegraded":"2023-12-24T19:00:22.544459+0000","last_fullsized":"2023-12-24T19:00:22.544459+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T00:30:36.804309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10075,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544271+0000","last_change":"2023-12-24T18:02:38.125769+0000","last_active":"2023-12-24T19:00:22.544271+0000","last_peered":"2023-12-24T19:00:22.544271+0000","last_clean":"2023-12-24T19:00:22.544271+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T19:00:22.544271+0000","last_undegraded":"2023-12-24T19:00:22.544271+0000","last_fullsized":"2023-12-24T19:00:22.544271+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T19:35:46.615178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3922,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.543550+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T19:00:22.543550+0000","last_peered":"2023-12-24T19:00:22.543550+0000","last_clean":"2023-12-24T19:00:22.543550+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T19:00:22.543550+0000","last_undegraded":"2023-12-24T19:00:22.543550+0000","last_fullsized":"2023-12-24T19:00:22.543550+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T19:39:13.450932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11203,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544205+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T19:00:22.544205+0000","last_peered":"2023-12-24T19:00:22.544205+0000","last_clean":"2023-12-24T19:00:22.544205+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T19:00:22.544205+0000","last_undegraded":"2023-12-24T19:00:22.544205+0000","last_fullsized":"2023-12-24T19:00:22.544205+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T01:58:57.653690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":284,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544525+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T19:00:22.544525+0000","last_peered":"2023-12-24T19:00:22.544525+0000","last_clean":"2023-12-24T19:00:22.544525+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T19:00:22.544525+0000","last_undegraded":"2023-12-24T19:00:22.544525+0000","last_fullsized":"2023-12-24T19:00:22.544525+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T22:45:34.173764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3529,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544114+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T19:00:22.544114+0000","last_peered":"2023-12-24T19:00:22.544114+0000","last_clean":"2023-12-24T19:00:22.544114+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T19:00:22.544114+0000","last_undegraded":"2023-12-24T19:00:22.544114+0000","last_fullsized":"2023-12-24T19:00:22.544114+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T23:52:54.059450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10731,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544559+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T19:00:22.544559+0000","last_peered":"2023-12-24T19:00:22.544559+0000","last_clean":"2023-12-24T19:00:22.544559+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T19:00:22.544559+0000","last_undegraded":"2023-12-24T19:00:22.544559+0000","last_fullsized":"2023-12-24T19:00:22.544559+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T19:02:38.676191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3502,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.543540+0000","last_change":"2023-12-24T18:02:38.129338+0000","last_active":"2023-12-24T19:00:22.543540+0000","last_peered":"2023-12-24T19:00:22.543540+0000","last_clean":"2023-12-24T19:00:22.543540+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T19:00:22.543540+0000","last_undegraded":"2023-12-24T19:00:22.543540+0000","last_fullsized":"2023-12-24T19:00:22.543540+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T22:55:41.056162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12783,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544395+0000","last_change":"2023-12-24T18:02:38.125565+0000","last_active":"2023-12-24T19:00:22.544395+0000","last_peered":"2023-12-24T19:00:22.544395+0000","last_clean":"2023-12-24T19:00:22.544395+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T19:00:22.544395+0000","last_undegraded":"2023-12-24T19:00:22.544395+0000","last_fullsized":"2023-12-24T19:00:22.544395+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T21:36:55.968107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3059,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544343+0000","last_change":"2023-12-24T18:02:38.124896+0000","last_active":"2023-12-24T19:00:22.544343+0000","last_peered":"2023-12-24T19:00:22.544343+0000","last_clean":"2023-12-24T19:00:22.544343+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T19:00:22.544343+0000","last_undegraded":"2023-12-24T19:00:22.544343+0000","last_fullsized":"2023-12-24T19:00:22.544343+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T00:37:18.060893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3313,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544290+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T19:00:22.544290+0000","last_peered":"2023-12-24T19:00:22.544290+0000","last_clean":"2023-12-24T19:00:22.544290+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T19:00:22.544290+0000","last_undegraded":"2023-12-24T19:00:22.544290+0000","last_fullsized":"2023-12-24T19:00:22.544290+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T01:45:50.432394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740570,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,2,2,6,12,4],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 20:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.20100000000000001,"5min":0.217,"15min":0.21299999999999999},"min":{"1min":0.155,"5min":0.155,"15min":0.14099999999999999},"max":{"1min":0.33100000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"last":0.17899999999999999},{"interface":"front","average":{"1min":0.22600000000000001,"5min":0.24299999999999999,"15min":0.23799999999999999},"min":{"1min":0.17199999999999999,"5min":0.16500000000000001,"15min":0.16300000000000001},"max":{"1min":0.36399999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"last":0.23499999999999999}]},{"osd":1,"last update":"Sun Dec 24 20:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.218,"5min":0.23400000000000001,"15min":0.23100000000000001},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.16400000000000001},"max":{"1min":0.34399999999999997,"5min":0.43099999999999999,"15min":0.46600000000000003},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.23999999999999999,"5min":0.26200000000000001,"15min":0.25900000000000001},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.40100000000000002,"5min":0.47599999999999998,"15min":0.49299999999999999},"last":0.20100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359740569,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 20:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.19500000000000001,"5min":0.20200000000000001,"15min":0.20399999999999999},"min":{"1min":0.154,"5min":0.13,"15min":0.127},"max":{"1min":0.33400000000000002,"5min":0.42699999999999999,"15min":0.46500000000000002},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.223,"5min":0.224,"15min":0.22700000000000001},"min":{"1min":0.16400000000000001,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.36499999999999999,"5min":0.41599999999999998,"15min":0.47599999999999998},"last":0.21199999999999999}]},{"osd":2,"last update":"Sun Dec 24 20:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.24299999999999999,"5min":0.245,"15min":0.24299999999999999},"min":{"1min":0.19500000000000001,"5min":0.16800000000000001,"15min":0.158},"max":{"1min":0.40200000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"last":0.247},{"interface":"front","average":{"1min":0.26900000000000002,"5min":0.27300000000000002,"15min":0.27200000000000002},"min":{"1min":0.19900000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.42599999999999999,"5min":0.47899999999999998,"15min":0.50600000000000001},"last":0.26600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359740571,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 20:59:55 2023","interfaces":[{"interface":"back","average":{"1min":0.19,"5min":0.20300000000000001,"15min":0.20000000000000001},"min":{"1min":0.14599999999999999,"5min":0.13600000000000001,"15min":0.13300000000000001},"max":{"1min":0.374,"5min":0.40000000000000002,"15min":0.41199999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":0.22500000000000001,"5min":0.22900000000000001,"15min":0.22700000000000001},"min":{"1min":0.17000000000000001,"5min":0.159,"15min":0.156},"max":{"1min":0.41099999999999998,"5min":0.45800000000000002,"15min":0.45800000000000002},"last":0.19600000000000001}]},{"osd":2,"last update":"Sun Dec 24 20:59:55 2023","interfaces":[{"interface":"back","average":{"1min":0.23499999999999999,"5min":0.24199999999999999,"15min":0.23899999999999999},"min":{"1min":0.17499999999999999,"5min":0.17299999999999999,"15min":0.16500000000000001},"max":{"1min":0.46100000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.254,"5min":0.26600000000000001,"15min":0.26200000000000001},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16700000000000001},"max":{"1min":0.443,"5min":0.45400000000000001,"15min":0.47799999999999998},"last":0.26200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:00:41.389 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:00:41.721 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:41.737+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:41.752 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:00:41.755 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:00:41.766 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":5649,"stamp":"2023-12-24T21:00:40.578839+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,2,2,6,12,516],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001844"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2824,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544349+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T19:00:22.544349+0000","last_peered":"2023-12-24T19:00:22.544349+0000","last_clean":"2023-12-24T19:00:22.544349+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T19:00:22.544349+0000","last_undegraded":"2023-12-24T19:00:22.544349+0000","last_fullsized":"2023-12-24T19:00:22.544349+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T20:39:55.726631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3618,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544409+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T19:00:22.544409+0000","last_peered":"2023-12-24T19:00:22.544409+0000","last_clean":"2023-12-24T19:00:22.544409+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T19:00:22.544409+0000","last_undegraded":"2023-12-24T19:00:22.544409+0000","last_fullsized":"2023-12-24T19:00:22.544409+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T01:55:17.265710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2981,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544459+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T19:00:22.544459+0000","last_peered":"2023-12-24T19:00:22.544459+0000","last_clean":"2023-12-24T19:00:22.544459+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T19:00:22.544459+0000","last_undegraded":"2023-12-24T19:00:22.544459+0000","last_fullsized":"2023-12-24T19:00:22.544459+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T00:30:36.804309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10075,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544271+0000","last_change":"2023-12-24T18:02:38.125769+0000","last_active":"2023-12-24T19:00:22.544271+0000","last_peered":"2023-12-24T19:00:22.544271+0000","last_clean":"2023-12-24T19:00:22.544271+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T19:00:22.544271+0000","last_undegraded":"2023-12-24T19:00:22.544271+0000","last_fullsized":"2023-12-24T19:00:22.544271+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T19:35:46.615178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3922,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.543550+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T19:00:22.543550+0000","last_peered":"2023-12-24T19:00:22.543550+0000","last_clean":"2023-12-24T19:00:22.543550+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T19:00:22.543550+0000","last_undegraded":"2023-12-24T19:00:22.543550+0000","last_fullsized":"2023-12-24T19:00:22.543550+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T19:39:13.450932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11203,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544205+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T19:00:22.544205+0000","last_peered":"2023-12-24T19:00:22.544205+0000","last_clean":"2023-12-24T19:00:22.544205+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T19:00:22.544205+0000","last_undegraded":"2023-12-24T19:00:22.544205+0000","last_fullsized":"2023-12-24T19:00:22.544205+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T01:58:57.653690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":284,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544525+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T19:00:22.544525+0000","last_peered":"2023-12-24T19:00:22.544525+0000","last_clean":"2023-12-24T19:00:22.544525+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T19:00:22.544525+0000","last_undegraded":"2023-12-24T19:00:22.544525+0000","last_fullsized":"2023-12-24T19:00:22.544525+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T22:45:34.173764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3529,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544114+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T19:00:22.544114+0000","last_peered":"2023-12-24T19:00:22.544114+0000","last_clean":"2023-12-24T19:00:22.544114+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T19:00:22.544114+0000","last_undegraded":"2023-12-24T19:00:22.544114+0000","last_fullsized":"2023-12-24T19:00:22.544114+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T23:52:54.059450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10731,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544559+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T19:00:22.544559+0000","last_peered":"2023-12-24T19:00:22.544559+0000","last_clean":"2023-12-24T19:00:22.544559+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T19:00:22.544559+0000","last_undegraded":"2023-12-24T19:00:22.544559+0000","last_fullsized":"2023-12-24T19:00:22.544559+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T19:02:38.676191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3502,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.543540+0000","last_change":"2023-12-24T18:02:38.129338+0000","last_active":"2023-12-24T19:00:22.543540+0000","last_peered":"2023-12-24T19:00:22.543540+0000","last_clean":"2023-12-24T19:00:22.543540+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T19:00:22.543540+0000","last_undegraded":"2023-12-24T19:00:22.543540+0000","last_fullsized":"2023-12-24T19:00:22.543540+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T22:55:41.056162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12783,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544395+0000","last_change":"2023-12-24T18:02:38.125565+0000","last_active":"2023-12-24T19:00:22.544395+0000","last_peered":"2023-12-24T19:00:22.544395+0000","last_clean":"2023-12-24T19:00:22.544395+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T19:00:22.544395+0000","last_undegraded":"2023-12-24T19:00:22.544395+0000","last_fullsized":"2023-12-24T19:00:22.544395+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T21:36:55.968107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3059,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544343+0000","last_change":"2023-12-24T18:02:38.124896+0000","last_active":"2023-12-24T19:00:22.544343+0000","last_peered":"2023-12-24T19:00:22.544343+0000","last_clean":"2023-12-24T19:00:22.544343+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T19:00:22.544343+0000","last_undegraded":"2023-12-24T19:00:22.544343+0000","last_fullsized":"2023-12-24T19:00:22.544343+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T00:37:18.060893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3313,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544290+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T19:00:22.544290+0000","last_peered":"2023-12-24T19:00:22.544290+0000","last_clean":"2023-12-24T19:00:22.544290+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T19:00:22.544290+0000","last_undegraded":"2023-12-24T19:00:22.544290+0000","last_fullsized":"2023-12-24T19:00:22.544290+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T01:45:50.432394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740570,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,2,2,6,12,4],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 20:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.20100000000000001,"5min":0.217,"15min":0.21299999999999999},"min":{"1min":0.155,"5min":0.155,"15min":0.14099999999999999},"max":{"1min":0.33100000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"last":0.17899999999999999},{"interface":"front","average":{"1min":0.22600000000000001,"5min":0.24299999999999999,"15min":0.23799999999999999},"min":{"1min":0.17199999999999999,"5min":0.16500000000000001,"15min":0.16300000000000001},"max":{"1min":0.36399999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"last":0.23499999999999999}]},{"osd":1,"last update":"Sun Dec 24 20:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.218,"5min":0.23400000000000001,"15min":0.23100000000000001},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.16400000000000001},"max":{"1min":0.34399999999999997,"5min":0.43099999999999999,"15min":0.46600000000000003},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.23999999999999999,"5min":0.26200000000000001,"15min":0.25900000000000001},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.40100000000000002,"5min":0.47599999999999998,"15min":0.49299999999999999},"last":0.20100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359740569,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 20:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.19500000000000001,"5min":0.20200000000000001,"15min":0.20399999999999999},"min":{"1min":0.154,"5min":0.13,"15min":0.127},"max":{"1min":0.33400000000000002,"5min":0.42699999999999999,"15min":0.46500000000000002},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.223,"5min":0.224,"15min":0.22700000000000001},"min":{"1min":0.16400000000000001,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.36499999999999999,"5min":0.41599999999999998,"15min":0.47599999999999998},"last":0.21199999999999999}]},{"osd":2,"last update":"Sun Dec 24 20:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.24299999999999999,"5min":0.245,"15min":0.24299999999999999},"min":{"1min":0.19500000000000001,"5min":0.16800000000000001,"15min":0.158},"max":{"1min":0.40200000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"last":0.247},{"interface":"front","average":{"1min":0.26900000000000002,"5min":0.27300000000000002,"15min":0.27200000000000002},"min":{"1min":0.19900000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.42599999999999999,"5min":0.47899999999999998,"15min":0.50600000000000001},"last":0.26600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359740571,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 20:59:55 2023","interfaces":[{"interface":"back","average":{"1min":0.19,"5min":0.20300000000000001,"15min":0.20000000000000001},"min":{"1min":0.14599999999999999,"5min":0.13600000000000001,"15min":0.13300000000000001},"max":{"1min":0.374,"5min":0.40000000000000002,"15min":0.41199999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":0.22500000000000001,"5min":0.22900000000000001,"15min":0.22700000000000001},"min":{"1min":0.17000000000000001,"5min":0.159,"15min":0.156},"max":{"1min":0.41099999999999998,"5min":0.45800000000000002,"15min":0.45800000000000002},"last":0.19600000000000001}]},{"osd":2,"last update":"Sun Dec 24 20:59:55 2023","interfaces":[{"interface":"back","average":{"1min":0.23499999999999999,"5min":0.24199999999999999,"15min":0.23899999999999999},"min":{"1min":0.17499999999999999,"5min":0.17299999999999999,"15min":0.16500000000000001},"max":{"1min":0.46100000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.254,"5min":0.26600000000000001,"15min":0.26200000000000001},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16700000000000001},"max":{"1min":0.443,"5min":0.45400000000000001,"15min":0.47799999999999998},"last":0.26200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:00:41.767 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-24T21:00:41.767 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:00:42.018 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:42.033+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:42.084 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:00:42.086 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:00:42.097 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":5649,"stamp":"2023-12-24T21:00:40.578839+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,2,2,6,12,516],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001844"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2824,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544349+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T19:00:22.544349+0000","last_peered":"2023-12-24T19:00:22.544349+0000","last_clean":"2023-12-24T19:00:22.544349+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T19:00:22.544349+0000","last_undegraded":"2023-12-24T19:00:22.544349+0000","last_fullsized":"2023-12-24T19:00:22.544349+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T20:39:55.726631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3618,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544409+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T19:00:22.544409+0000","last_peered":"2023-12-24T19:00:22.544409+0000","last_clean":"2023-12-24T19:00:22.544409+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T19:00:22.544409+0000","last_undegraded":"2023-12-24T19:00:22.544409+0000","last_fullsized":"2023-12-24T19:00:22.544409+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T01:55:17.265710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2981,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544459+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T19:00:22.544459+0000","last_peered":"2023-12-24T19:00:22.544459+0000","last_clean":"2023-12-24T19:00:22.544459+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T19:00:22.544459+0000","last_undegraded":"2023-12-24T19:00:22.544459+0000","last_fullsized":"2023-12-24T19:00:22.544459+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T00:30:36.804309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10075,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544271+0000","last_change":"2023-12-24T18:02:38.125769+0000","last_active":"2023-12-24T19:00:22.544271+0000","last_peered":"2023-12-24T19:00:22.544271+0000","last_clean":"2023-12-24T19:00:22.544271+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T19:00:22.544271+0000","last_undegraded":"2023-12-24T19:00:22.544271+0000","last_fullsized":"2023-12-24T19:00:22.544271+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T19:35:46.615178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3922,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.543550+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T19:00:22.543550+0000","last_peered":"2023-12-24T19:00:22.543550+0000","last_clean":"2023-12-24T19:00:22.543550+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T19:00:22.543550+0000","last_undegraded":"2023-12-24T19:00:22.543550+0000","last_fullsized":"2023-12-24T19:00:22.543550+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T19:39:13.450932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11203,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544205+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T19:00:22.544205+0000","last_peered":"2023-12-24T19:00:22.544205+0000","last_clean":"2023-12-24T19:00:22.544205+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T19:00:22.544205+0000","last_undegraded":"2023-12-24T19:00:22.544205+0000","last_fullsized":"2023-12-24T19:00:22.544205+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T01:58:57.653690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":284,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544525+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T19:00:22.544525+0000","last_peered":"2023-12-24T19:00:22.544525+0000","last_clean":"2023-12-24T19:00:22.544525+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T19:00:22.544525+0000","last_undegraded":"2023-12-24T19:00:22.544525+0000","last_fullsized":"2023-12-24T19:00:22.544525+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T22:45:34.173764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3529,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544114+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T19:00:22.544114+0000","last_peered":"2023-12-24T19:00:22.544114+0000","last_clean":"2023-12-24T19:00:22.544114+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T19:00:22.544114+0000","last_undegraded":"2023-12-24T19:00:22.544114+0000","last_fullsized":"2023-12-24T19:00:22.544114+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T23:52:54.059450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10731,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544559+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T19:00:22.544559+0000","last_peered":"2023-12-24T19:00:22.544559+0000","last_clean":"2023-12-24T19:00:22.544559+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T19:00:22.544559+0000","last_undegraded":"2023-12-24T19:00:22.544559+0000","last_fullsized":"2023-12-24T19:00:22.544559+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T19:02:38.676191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3502,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.543540+0000","last_change":"2023-12-24T18:02:38.129338+0000","last_active":"2023-12-24T19:00:22.543540+0000","last_peered":"2023-12-24T19:00:22.543540+0000","last_clean":"2023-12-24T19:00:22.543540+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T19:00:22.543540+0000","last_undegraded":"2023-12-24T19:00:22.543540+0000","last_fullsized":"2023-12-24T19:00:22.543540+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T22:55:41.056162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12783,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544395+0000","last_change":"2023-12-24T18:02:38.125565+0000","last_active":"2023-12-24T19:00:22.544395+0000","last_peered":"2023-12-24T19:00:22.544395+0000","last_clean":"2023-12-24T19:00:22.544395+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T19:00:22.544395+0000","last_undegraded":"2023-12-24T19:00:22.544395+0000","last_fullsized":"2023-12-24T19:00:22.544395+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T21:36:55.968107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3059,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544343+0000","last_change":"2023-12-24T18:02:38.124896+0000","last_active":"2023-12-24T19:00:22.544343+0000","last_peered":"2023-12-24T19:00:22.544343+0000","last_clean":"2023-12-24T19:00:22.544343+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T19:00:22.544343+0000","last_undegraded":"2023-12-24T19:00:22.544343+0000","last_fullsized":"2023-12-24T19:00:22.544343+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T00:37:18.060893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3313,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544290+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T19:00:22.544290+0000","last_peered":"2023-12-24T19:00:22.544290+0000","last_clean":"2023-12-24T19:00:22.544290+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T19:00:22.544290+0000","last_undegraded":"2023-12-24T19:00:22.544290+0000","last_fullsized":"2023-12-24T19:00:22.544290+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T01:45:50.432394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740570,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,2,2,6,12,4],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 20:59:35 2023","interfaces":[{"interface":"back","average":{"1min":0.20100000000000001,"5min":0.217,"15min":0.21299999999999999},"min":{"1min":0.155,"5min":0.155,"15min":0.14099999999999999},"max":{"1min":0.33100000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"last":0.17899999999999999},{"interface":"front","average":{"1min":0.22600000000000001,"5min":0.24299999999999999,"15min":0.23799999999999999},"min":{"1min":0.17199999999999999,"5min":0.16500000000000001,"15min":0.16300000000000001},"max":{"1min":0.36399999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"last":0.23499999999999999}]},{"osd":1,"last update":"Sun Dec 24 20:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.218,"5min":0.23400000000000001,"15min":0.23100000000000001},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.16400000000000001},"max":{"1min":0.34399999999999997,"5min":0.43099999999999999,"15min":0.46600000000000003},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.23999999999999999,"5min":0.26200000000000001,"15min":0.25900000000000001},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.40100000000000002,"5min":0.47599999999999998,"15min":0.49299999999999999},"last":0.20100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359740569,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 20:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.19500000000000001,"5min":0.20200000000000001,"15min":0.20399999999999999},"min":{"1min":0.154,"5min":0.13,"15min":0.127},"max":{"1min":0.33400000000000002,"5min":0.42699999999999999,"15min":0.46500000000000002},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.223,"5min":0.224,"15min":0.22700000000000001},"min":{"1min":0.16400000000000001,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.36499999999999999,"5min":0.41599999999999998,"15min":0.47599999999999998},"last":0.21199999999999999}]},{"osd":2,"last update":"Sun Dec 24 20:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.24299999999999999,"5min":0.245,"15min":0.24299999999999999},"min":{"1min":0.19500000000000001,"5min":0.16800000000000001,"15min":0.158},"max":{"1min":0.40200000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"last":0.247},{"interface":"front","average":{"1min":0.26900000000000002,"5min":0.27300000000000002,"15min":0.27200000000000002},"min":{"1min":0.19900000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.42599999999999999,"5min":0.47899999999999998,"15min":0.50600000000000001},"last":0.26600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359740571,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 20:59:55 2023","interfaces":[{"interface":"back","average":{"1min":0.19,"5min":0.20300000000000001,"15min":0.20000000000000001},"min":{"1min":0.14599999999999999,"5min":0.13600000000000001,"15min":0.13300000000000001},"max":{"1min":0.374,"5min":0.40000000000000002,"15min":0.41199999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":0.22500000000000001,"5min":0.22900000000000001,"15min":0.22700000000000001},"min":{"1min":0.17000000000000001,"5min":0.159,"15min":0.156},"max":{"1min":0.41099999999999998,"5min":0.45800000000000002,"15min":0.45800000000000002},"last":0.19600000000000001}]},{"osd":2,"last update":"Sun Dec 24 20:59:55 2023","interfaces":[{"interface":"back","average":{"1min":0.23499999999999999,"5min":0.24199999999999999,"15min":0.23899999999999999},"min":{"1min":0.17499999999999999,"5min":0.17299999999999999,"15min":0.16500000000000001},"max":{"1min":0.46100000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.254,"5min":0.26600000000000001,"15min":0.26200000000000001},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.16700000000000001},"max":{"1min":0.443,"5min":0.45400000000000001,"15min":0.47799999999999998},"last":0.26200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:00:42.098 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-24T21:00:42.183 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:42.199+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:42.559 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:00:42.560 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":162,"fsid":"0f2278f5-9601-4df5-8109-682fbac125ff","created":"2023-12-24T17:57:27.057964+0000","modified":"2023-12-24T19:00:52.539743+0000","last_up_change":"2023-12-24T17:57:36.716834+0000","last_in_change":"2023-12-24T17:57:28.019393+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-24T17:57:36.963424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-24T17:57:42.305786+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"159","last_force_op_resend":"18","last_force_op_resend_prenautilus":"18","last_force_op_resend_preluminous":"18","auid":0,"snap_mode":"selfmanaged","snap_seq":93,"snap_epoch":159,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[3],"tier_of":-1,"read_tier":3,"write_tier":3,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cache","create_time":"2023-12-24T17:57:49.828478+0000","flags":8201,"flags_names":"hashpspool,incomplete_clones,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"159","last_force_op_resend":"18","last_force_op_resend_prenautilus":"18","last_force_op_resend_preluminous":"18","auid":0,"snap_mode":"selfmanaged","snap_seq":93,"snap_epoch":159,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":2,"read_tier":-1,"write_tier":-1,"cache_mode":"writeback","target_max_bytes":0,"target_max_objects":250,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"bloom","false_positive_probability":0.050000000000000003,"target_size":0,"seed":0},"hit_set_period":60,"hit_set_count":8,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[1000000,1000000,1000000,1000000,1000000,1000000,1000000,1000000],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2c99abb1-7a79-494e-a998-e2942573c6ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":114598},{"type":"v1","addr":"172.21.15.87:6817","nonce":114598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":114598},{"type":"v1","addr":"172.21.15.87:6819","nonce":114598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":114598},{"type":"v1","addr":"172.21.15.87:6823","nonce":114598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":114598},{"type":"v1","addr":"172.21.15.87:6821","nonce":114598}]},"public_addr":"172.21.15.87:6817/114598","cluster_addr":"172.21.15.87:6819/114598","heartbeat_back_addr":"172.21.15.87:6823/114598","heartbeat_front_addr":"172.21.15.87:6821/114598","state":["exists","up"]},{"osd":1,"uuid":"2f9896f4-15d2-40c9-8bd7-05fe1e6d8b61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":114597},{"type":"v1","addr":"172.21.15.87:6809","nonce":114597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":114597},{"type":"v1","addr":"172.21.15.87:6811","nonce":114597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":114597},{"type":"v1","addr":"172.21.15.87:6815","nonce":114597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":114597},{"type":"v1","addr":"172.21.15.87:6813","nonce":114597}]},"public_addr":"172.21.15.87:6809/114597","cluster_addr":"172.21.15.87:6811/114597","heartbeat_back_addr":"172.21.15.87:6815/114597","heartbeat_front_addr":"172.21.15.87:6813/114597","state":["exists","up"]},{"osd":2,"uuid":"0403cd42-78bb-43a2-9809-62b634e0e5df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":114582},{"type":"v1","addr":"172.21.15.87:6801","nonce":114582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":114582},{"type":"v1","addr":"172.21.15.87:6803","nonce":114582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":114582},{"type":"v1","addr":"172.21.15.87:6807","nonce":114582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":114582},{"type":"v1","addr":"172.21.15.87:6805","nonce":114582}]},"public_addr":"172.21.15.87:6801/114582","cluster_addr":"172.21.15.87:6803/114582","heartbeat_back_addr":"172.21.15.87:6807/114582","heartbeat_front_addr":"172.21.15.87:6805/114582","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-24T17:57:33.148164+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-24T17:57:33.220152+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-24T17:57:33.135613+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":4,"length":1}]},{"pool":5,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-24T21:00:42.697 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:42.711+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:42.979 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:42.995+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:43.228 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:43.244+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:43.575 INFO:tasks.ceph:Scrubbing osd.0 2023-12-24T21:00:43.576 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-12-24T21:00:43.728 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:43.744+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:43.996 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:44.012+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:44.186 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:44.202+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:44.766 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:44.782+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:45.040 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:45.056+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:45.180 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:45.196+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:45.791 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:45.807+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:46.086 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:46.102+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:46.203 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:46.219+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:46.752 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:46.767+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:47.099 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:47.115+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:47.225 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:47.241+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:47.792 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:47.807+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:48.105 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:48.120+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:48.224 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:48.240+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:48.770 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:48.786+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:49.131 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:49.147+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:49.273 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:49.289+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:49.787 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:49.803+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:50.168 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:50.183+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:50.263 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:50.279+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:50.792 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:50.808+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:51.164 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:51.180+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:51.314 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:51.330+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:51.800 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:51.815+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:52.198 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:52.213+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:52.353 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:52.369+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:52.767 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:52.784+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:53.206 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:53.222+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:53.349 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:53.365+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:53.739 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:53.755+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:54.160 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:54.176+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:54.379 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:54.396+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:54.764 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:54.780+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:55.155 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:55.172+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:55.359 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:55.376+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:55.797 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:55.813+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:56.116 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:56.132+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:56.394 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:56.410+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:56.771 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:56.787+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:57.079 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:57.095+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:57.387 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:57.404+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:57.801 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:57.818+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:58.113 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:58.130+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:58.416 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:58.432+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:58.825 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:58.841+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:59.155 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:00:59.172+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:59.428 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:00:59.444+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:00:59.783 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:00:59.799+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:00.149 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:00.166+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:00.455 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:00.472+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:00.795 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:00.812+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:01.174 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:01.190+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:01.502 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:01.519+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:01.823 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:01.839+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:02.138 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:02.155+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:02.489 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:02.505+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:02.823 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:02.840+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:03.096 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:03.112+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:03.485 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:03.501+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:03.854 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:03.870+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:04.083 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:04.099+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:04.518 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:04.534+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:04.893 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:04.909+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:05.060 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:05.076+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:05.564 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:05.580+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:05.868 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:05.884+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:06.045 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:06.061+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:06.525 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:06.541+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:06.883 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:06.899+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:06.996 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:07.012+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:07.482 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:07.497+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:07.848 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:07.863+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:08.016 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:08.031+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:08.497 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:08.512+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:08.893 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:08.908+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:08.984 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:08.999+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:09.530 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:09.545+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:09.858 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:09.873+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:10.020 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:10.035+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:10.571 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:10.586+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:10.811 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:10.826+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:11.043 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:11.058+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:11.596 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:11.611+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:11.830 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:11.845+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:12.084 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:12.099+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:12.601 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:12.615+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:12.842 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:12.857+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:13.124 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:13.139+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:13.628 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:13.643+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:13.827 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:13.842+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:14.077 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:14.092+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:14.667 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:14.681+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:14.806 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:14.821+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:15.076 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:15.091+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:15.634 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:15.649+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:15.764 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:15.779+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:16.082 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:16.097+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:16.636 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:16.651+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:16.813 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:16.828+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:17.050 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:17.064+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:17.638 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:17.653+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:17.845 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:17.860+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:18.036 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:18.051+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:18.622 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:18.636+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:18.815 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:18.830+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:18.987 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:19.002+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:19.590 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:19.605+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:19.791 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:19.806+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:19.992 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:20.007+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:20.637 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:20.652+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:20.802 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:20.816+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:20.992 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:21.007+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:21.668 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:21.683+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:21.807 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:21.821+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:21.996 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:22.011+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:22.638 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:22.653+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:22.820 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:22.835+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:23.046 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:23.061+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:23.603 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:23.617+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:23.840 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:23.855+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:24.065 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:24.080+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:24.609 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:24.623+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:24.796 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:24.810+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:25.046 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:25.060+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:25.603 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:25.618+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:25.808 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:25.823+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:26.020 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:26.034+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:26.616 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:26.630+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:26.790 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:26.805+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:27.019 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:27.034+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:27.576 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:27.591+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:27.823 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:27.837+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:28.041 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:28.055+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:28.616 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:28.630+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:28.854 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:28.869+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:29.081 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:29.095+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:29.608 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:29.622+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:29.871 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:29.885+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:30.047 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:30.061+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:30.628 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:30.642+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:30.882 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:30.896+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:31.096 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:31.111+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:31.655 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:31.670+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:31.919 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:31.933+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:32.051 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:32.066+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:32.668 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:32.683+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:32.920 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:32.934+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:33.066 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:33.080+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:33.671 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:33.685+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:33.920 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:33.934+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:34.024 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:34.038+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:34.683 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:34.697+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:34.906 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:34.920+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:35.073 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:35.087+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:35.678 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:35.692+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:35.901 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:35.916+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:36.046 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:36.061+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:36.682 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:36.696+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:36.949 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:36.963+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:37.060 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:37.074+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:37.659 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:37.673+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:37.989 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:38.003+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:38.021 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:38.035+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:38.702 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:38.717+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:39.027 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:39.042+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:39.028 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:39.043+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:39.665 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:39.679+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:40.011 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:40.026+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:40.061 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:40.076+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:40.650 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:40.665+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:40.981 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:40.996+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:41.046 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:41.061+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:41.655 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:41.670+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:41.997 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:42.012+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:42.059 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:42.074+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:42.660 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:42.675+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:43.046 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:43.060+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:43.104 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:43.119+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:43.632 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:43.647+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:44.052 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:44.067+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:44.077 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:44.092+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:44.626 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:44.641+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:45.053 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:45.068+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:45.109 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:45.124+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:45.676 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:45.690+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:46.047 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:46.062+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:46.112 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:46.127+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:46.686 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:46.701+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:47.005 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:47.020+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:47.103 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:47.118+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:47.697 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:47.712+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:48.016 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:48.030+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:48.085 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:48.100+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:48.678 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:48.692+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:48.971 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:48.986+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:49.131 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:49.146+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:49.728 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:49.742+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:49.992 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:50.007+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:50.086 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:50.101+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:50.747 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:50.762+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:51.012 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:51.027+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:51.116 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:51.131+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:51.739 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:51.754+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:51.966 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:51.980+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:52.106 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:52.120+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:52.728 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:52.741+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:52.943 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:52.956+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:53.070 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:53.084+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:53.737 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:53.751+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:53.951 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:53.964+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:54.117 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:54.131+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:54.725 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:54.739+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:54.971 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:54.985+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:55.098 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:55.112+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:55.713 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:55.726+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:55.994 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:56.007+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:56.096 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:56.110+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:56.673 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:56.687+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:56.979 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:56.993+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:57.076 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:57.090+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:57.653 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:57.665+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:57.935 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:57.949+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:58.100 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:58.114+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:58.613 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:58.627+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:58.968 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:58.981+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:59.063 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:01:59.077+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:59.608 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:01:59.621+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:01:59.927 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:01:59.941+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:00.112 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:00.126+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:00.617 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:00.631+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:00.918 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:00.931+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:01.157 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:01.171+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:01.604 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:01.617+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:01.902 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:01.916+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:02.185 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:02.199+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:02.608 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:02.621+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:02.885 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:02.899+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:03.207 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:03.220+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:03.564 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:03.577+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:03.934 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:03.947+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:04.229 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:04.242+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:04.580 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:04.593+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:04.954 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:04.967+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:05.257 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:05.270+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:05.538 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:05.551+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:05.970 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:05.983+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:06.254 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:06.268+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:06.581 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:06.595+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:06.983 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:06.997+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:07.262 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:07.275+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:07.577 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:07.590+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:07.961 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:07.974+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:08.213 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:08.226+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:08.570 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:08.583+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:08.979 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:08.992+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:09.253 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:09.266+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:09.524 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:09.537+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:09.948 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:09.961+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:10.244 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:10.257+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:10.493 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:10.506+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:10.954 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:10.967+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:11.255 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:11.268+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:11.521 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:11.534+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:11.921 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:11.934+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:12.262 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:12.275+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:12.497 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:12.510+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:12.888 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:12.901+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:13.243 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:13.256+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:13.499 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:13.513+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:13.864 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:13.877+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:14.232 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:14.245+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:14.534 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:14.547+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:14.859 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:14.872+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:15.268 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:15.281+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:15.521 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:15.535+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:15.882 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:15.895+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:16.253 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:16.266+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:16.513 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:16.526+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:16.916 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:16.929+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:17.283 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:17.297+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:17.523 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:17.536+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:17.884 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:17.897+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:18.307 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:18.321+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:18.496 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:18.509+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:18.875 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:18.888+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:19.267 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:19.280+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:19.476 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:19.489+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:19.910 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:19.923+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:20.224 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:20.237+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:20.502 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:20.515+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:20.925 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:20.938+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:21.193 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:21.205+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:21.470 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:21.483+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:21.942 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:21.954+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:22.234 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:22.247+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:22.472 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:22.484+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:22.945 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:22.957+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:23.266 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:23.279+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:23.464 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:23.478+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:23.911 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:23.925+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:24.284 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:24.298+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:24.440 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:24.454+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:24.898 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:24.912+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:25.331 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:25.345+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:25.473 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:25.487+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:25.909 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:25.922+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:26.365 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:26.379+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:26.462 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:26.476+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:26.929 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:26.943+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:27.403 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:27.416+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:27.502 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:27.516+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:27.940 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:27.953+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:28.443 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:28.457+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:28.464 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:28.478+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:28.974 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:28.987+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:29.434 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:29.448+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:29.455 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:29.468+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:29.991 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:30.004+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:30.452 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:30.465+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:30.487 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:30.500+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:31.040 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:31.054+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:31.497 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:31.510+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:31.524 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:31.538+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:32.060 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:32.073+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:32.534 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:32.547+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:32.540 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:32.554+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:33.085 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:33.098+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:33.491 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:33.504+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:33.528 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:33.542+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:34.052 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:34.065+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:34.503 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:34.517+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:34.515 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:34.529+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:35.088 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:35.101+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:35.490 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:35.503+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:35.544 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:35.557+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:36.112 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:36.126+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:36.454 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:36.468+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:36.515 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:36.528+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:37.140 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:37.153+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:37.442 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:37.455+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:37.516 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:37.529+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:38.125 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:38.137+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:38.448 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:38.461+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:38.506 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:38.518+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:39.131 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:39.143+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:39.445 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:39.457+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:39.489 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:39.501+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:40.119 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:40.131+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:40.451 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:40.463+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:40.464 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:40.476+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:41.110 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:41.122+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:41.460 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:41.473+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:41.499 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:41.511+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:42.102 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:42.114+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:42.481 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:42.493+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:42.549 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:42.561+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:43.127 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:43.139+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:43.527 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:43.539+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:43.591 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:43.603+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:43.610 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-24T21:02:43.610 INFO:tasks.ceph:Scrubbing osd.1 2023-12-24T21:02:43.611 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-12-24T21:02:44.140 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:44.152+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:44.502 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:44.514+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:44.568 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:44.580+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:45.128 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:45.140+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:45.550 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:45.562+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:45.602 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:45.614+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:46.153 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:46.165+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:46.530 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:46.542+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:46.596 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:46.608+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:47.197 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:47.209+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:47.514 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:47.526+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:47.566 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:47.578+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:48.222 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:48.234+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:48.561 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:48.573+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:48.567 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:48.579+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:49.237 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:49.249+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:49.562 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:49.574+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:49.579 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:49.591+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:50.204 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:50.216+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:50.524 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:50.536+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:50.619 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:50.631+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:51.221 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:51.232+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:51.521 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:51.533+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:51.646 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:51.658+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:52.246 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:52.258+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:52.522 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:52.534+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:52.669 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:52.681+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:53.286 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:53.298+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:53.508 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:53.520+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:53.715 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:53.727+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:54.283 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:54.295+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:54.482 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:54.494+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:54.668 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:54.680+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:55.291 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:55.303+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:55.444 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:55.456+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:55.642 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:55.654+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:56.278 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:56.290+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:56.399 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:56.411+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:56.613 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:56.625+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:57.265 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:57.277+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:57.384 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:57.395+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:57.600 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:57.612+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:58.256 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:58.268+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:58.387 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:58.398+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:58.610 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:58.621+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:59.282 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:02:59.294+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:59.375 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:02:59.387+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:02:59.577 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:02:59.589+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:00.333 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:00.344+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:00.410 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:00.421+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:00.536 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:00.548+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:01.299 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:01.311+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:01.426 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:01.438+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:01.557 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:01.569+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:02.345 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:02.357+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:02.434 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:02.445+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:02.522 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:02.533+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:03.340 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:03.351+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:03.472 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:03.484+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:03.555 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:03.567+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:04.344 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:04.356+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:04.445 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:04.456+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:04.509 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:04.521+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:05.326 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:05.338+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:05.451 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:05.463+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:05.547 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:05.559+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:06.325 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:06.337+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:06.461 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:06.473+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:06.568 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:06.580+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:07.311 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:07.323+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:07.413 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:07.424+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:07.527 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:07.539+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:08.321 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:08.332+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:08.421 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:08.434+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:08.543 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:08.555+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:09.361 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:09.373+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:09.407 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:09.419+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:09.545 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:09.558+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:10.342 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:10.355+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:10.412 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:10.425+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:10.564 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:10.576+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:11.333 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:11.346+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:11.449 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:11.462+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:11.563 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:11.576+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:12.285 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:12.297+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:12.497 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:12.509+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:12.561 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:12.573+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:13.240 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:13.253+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:13.487 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:13.499+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:13.530 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:13.542+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:14.276 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:14.288+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:14.488 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:14.500+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:14.552 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:14.564+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:15.299 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:15.312+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:15.445 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:15.458+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:15.520 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:15.532+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:16.254 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:16.266+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:16.402 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:16.414+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:16.519 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:16.531+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:17.267 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:17.280+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:17.435 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:17.447+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:17.470 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:17.482+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:18.280 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:18.292+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:18.428 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:18.440+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:18.441 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:18.453+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:19.274 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:19.286+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:19.426 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:19.438+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:19.468 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:19.480+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:20.300 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:20.312+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:20.377 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:20.389+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:20.436 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:20.448+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:21.315 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:21.327+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:21.382 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:21.394+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:21.432 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:21.444+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:22.307 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:22.318+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:22.426 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:22.437+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:22.452 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:22.463+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:23.326 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:23.337+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:23.395 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:23.406+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:23.492 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:23.502+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:24.343 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:24.354+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:24.368 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:24.379+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:24.512 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:24.521+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:25.366 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:25.377+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:25.370 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:25.381+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:25.509 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:25.520+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:26.337 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:26.348+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:26.394 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:26.405+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:26.497 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:26.508+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:27.331 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:27.342+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:27.360 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:27.371+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:27.495 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:27.506+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:28.353 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:28.364+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:28.361 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:28.372+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:28.455 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:28.466+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:29.314 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:29.325+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:29.385 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:29.396+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:29.462 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:29.473+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:30.290 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:30.300+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:30.423 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:30.434+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:30.437 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:30.448+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:31.299 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:31.310+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:31.415 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:31.426+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:31.432 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:31.443+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:32.319 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:32.329+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:32.464 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:32.475+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:32.470 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:32.481+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:33.323 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:33.334+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:33.477 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:33.488+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:33.484 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:33.495+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:34.365 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:34.376+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:34.452 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:34.463+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:34.527 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:34.538+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:35.346 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:35.356+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:35.440 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:35.451+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:35.554 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:35.564+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:36.382 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:36.393+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:36.436 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:36.445+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:36.600 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:36.610+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:37.406 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:37.416+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:37.433 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:37.443+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:37.627 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:37.638+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:38.393 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:38.403+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:38.437 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:38.448+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:38.672 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:38.682+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:39.406 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:39.417+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:39.419 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:39.430+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:39.708 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:39.718+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:40.373 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:40.383+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:40.447 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:40.457+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:40.686 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:40.697+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:41.384 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:41.394+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:41.444 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:41.455+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:41.736 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:41.747+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:42.373 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:42.384+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:42.399 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:42.409+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:42.755 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:42.765+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:43.355 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:43.365+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:43.382 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:43.392+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:43.719 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:43.730+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:44.389 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:44.400+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:44.422 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:44.433+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:44.707 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:44.717+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:45.370 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:45.381+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:45.393 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:45.404+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:45.691 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:45.701+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:46.349 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:46.359+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:46.373 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:46.383+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:46.699 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:46.709+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:47.340 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:47.351+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:47.395 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:47.405+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:47.684 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:47.695+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:48.328 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:48.338+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:48.379 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:48.390+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:48.684 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:48.694+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:49.320 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:49.330+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:49.337 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:49.347+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:49.669 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:49.679+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:50.303 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:50.314+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:50.359 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:50.370+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:50.645 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:50.655+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:51.312 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:51.323+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:51.404 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:51.415+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:51.694 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:51.704+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:52.270 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:52.281+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:52.393 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:52.403+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:52.720 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:52.730+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:53.290 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:53.301+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:53.374 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:53.386+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:53.719 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:53.731+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:54.244 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:54.256+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:54.362 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:54.373+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:54.711 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:54.722+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:55.210 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:55.222+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:55.354 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:55.365+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:55.738 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:55.749+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:56.252 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:56.264+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:56.312 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:56.323+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:56.704 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:56.715+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:57.281 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:57.292+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:57.317 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:57.328+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:57.738 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:57.749+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:58.274 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:58.286+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:58.285 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:58.296+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:58.693 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:58.705+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:59.260 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:03:59.271+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:59.323 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:03:59.334+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:03:59.701 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:03:59.712+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:00.283 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:00.294+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:00.343 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:00.354+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:00.674 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:00.685+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:01.272 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:01.282+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:01.376 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:01.387+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:01.672 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:01.683+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:02.230 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:02.241+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:02.395 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:02.406+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:02.645 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:02.656+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:03.196 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:03.207+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:03.417 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:03.428+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:03.599 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:03.610+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:04.242 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:04.253+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:04.441 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:04.452+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:04.598 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:04.609+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:05.199 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:05.210+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:05.451 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:05.462+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:05.612 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:05.623+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:06.230 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:06.241+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:06.410 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:06.420+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:06.637 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:06.647+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:07.259 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:07.269+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:07.412 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:07.422+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:07.682 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:07.692+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:08.289 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:08.299+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:08.422 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:08.432+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:08.662 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:08.672+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:09.279 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:09.289+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:09.383 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:09.393+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:09.701 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:09.711+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:10.271 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:10.280+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:10.432 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:10.442+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:10.728 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:10.738+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:11.234 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:11.244+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:11.440 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:11.450+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:11.691 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:11.701+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:12.209 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:12.219+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:12.478 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:12.488+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:12.734 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:12.744+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:13.222 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:13.232+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:13.477 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:13.487+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:13.697 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:13.707+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:14.191 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:14.200+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:14.494 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:14.504+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:14.667 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:14.677+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:15.164 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:15.174+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:15.513 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:15.522+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:15.707 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:15.717+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:16.154 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:16.163+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:16.520 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:16.530+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:16.671 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:16.681+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:17.177 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:17.187+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:17.545 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:17.555+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:17.718 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:17.727+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:18.132 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:18.141+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:18.525 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:18.534+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:18.757 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:18.766+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:19.179 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:19.188+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:19.541 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:19.550+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:19.712 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:19.722+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:20.211 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:20.221+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:20.550 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:20.560+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:20.677 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:20.687+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:21.247 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:21.257+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:21.547 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:21.556+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:21.712 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:21.722+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:22.253 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:22.262+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:22.592 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:22.602+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:22.730 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:22.740+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:23.230 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:23.239+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:23.581 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:23.591+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:23.687 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:23.697+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:24.195 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:24.205+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:24.570 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:24.579+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:24.684 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:24.694+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:25.209 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:25.218+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:25.601 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:25.611+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:25.710 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:25.720+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:26.221 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:26.230+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:26.644 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:26.653+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:26.744 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:26.754+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:27.205 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:27.214+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:27.679 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:27.688+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:27.755 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:27.764+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:28.240 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:28.250+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:28.633 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:28.642+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:28.743 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:28.753+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:29.248 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:29.258+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:29.642 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:29.652+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:29.761 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:29.771+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:30.202 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:30.212+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:30.674 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:30.683+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:30.787 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:30.797+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:31.160 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:31.169+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:31.666 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:31.675+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:31.743 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:31.752+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:32.121 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:32.131+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:32.694 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:32.703+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:32.713 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:32.722+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:33.094 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:33.103+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:33.677 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:33.686+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:33.758 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:33.768+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:34.111 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:34.120+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:34.713 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:34.722+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:34.719 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:34.728+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:35.084 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:35.093+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:35.710 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:35.720+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:35.741 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:35.751+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:36.132 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:36.141+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:36.682 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:36.691+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:36.695 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:36.705+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:37.149 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:37.159+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:37.657 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:37.666+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:37.687 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:37.697+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:38.192 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:38.202+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:38.659 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:38.669+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:38.697 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:38.708+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:39.200 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:39.210+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:39.695 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:39.705+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:39.738 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:39.748+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:40.212 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:40.222+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:40.694 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:40.704+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:40.708 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:40.718+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:41.240 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:41.250+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:41.678 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:41.688+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:41.739 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:41.750+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:42.255 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:42.265+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:42.727 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:42.737+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:42.788 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:42.798+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:43.269 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:43.279+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:43.646 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-24T21:04:43.646 INFO:tasks.ceph:Scrubbing osd.2 2023-12-24T21:04:43.647 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-12-24T21:04:43.752 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:43.762+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:43.776 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:43.786+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:43.789 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2023-12-24T21:04:43.789 INFO:teuthology.orchestra.run.smithi087.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-12-24T21:04:43.789 INFO:teuthology.orchestra.run.smithi087.stdout:} 2023-12-24T21:04:43.802 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-12-24T21:04:44.160 INFO:teuthology.orchestra.run.smithi087.stderr:instructed osd(s) 2 to deep-scrub 2023-12-24T21:04:44.172 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:04:44.229 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:44.239+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:44.516 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:04:44.517 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:04:44.528 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":5770,"stamp":"2023-12-24T21:04:42.614155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,2,6,12,516],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001761"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2824,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544349+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T19:00:22.544349+0000","last_peered":"2023-12-24T19:00:22.544349+0000","last_clean":"2023-12-24T19:00:22.544349+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T19:00:22.544349+0000","last_undegraded":"2023-12-24T19:00:22.544349+0000","last_fullsized":"2023-12-24T19:00:22.544349+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T20:39:55.726631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3618,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544409+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T19:00:22.544409+0000","last_peered":"2023-12-24T19:00:22.544409+0000","last_clean":"2023-12-24T19:00:22.544409+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T19:00:22.544409+0000","last_undegraded":"2023-12-24T19:00:22.544409+0000","last_fullsized":"2023-12-24T19:00:22.544409+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T01:55:17.265710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2981,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544459+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T19:00:22.544459+0000","last_peered":"2023-12-24T19:00:22.544459+0000","last_clean":"2023-12-24T19:00:22.544459+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T19:00:22.544459+0000","last_undegraded":"2023-12-24T19:00:22.544459+0000","last_fullsized":"2023-12-24T19:00:22.544459+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T00:30:36.804309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10075,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544271+0000","last_change":"2023-12-24T18:02:38.125769+0000","last_active":"2023-12-24T19:00:22.544271+0000","last_peered":"2023-12-24T19:00:22.544271+0000","last_clean":"2023-12-24T19:00:22.544271+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T19:00:22.544271+0000","last_undegraded":"2023-12-24T19:00:22.544271+0000","last_fullsized":"2023-12-24T19:00:22.544271+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T19:35:46.615178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3922,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.543550+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T19:00:22.543550+0000","last_peered":"2023-12-24T19:00:22.543550+0000","last_clean":"2023-12-24T19:00:22.543550+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T19:00:22.543550+0000","last_undegraded":"2023-12-24T19:00:22.543550+0000","last_fullsized":"2023-12-24T19:00:22.543550+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T19:39:13.450932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11203,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544205+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T19:00:22.544205+0000","last_peered":"2023-12-24T19:00:22.544205+0000","last_clean":"2023-12-24T19:00:22.544205+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T19:00:22.544205+0000","last_undegraded":"2023-12-24T19:00:22.544205+0000","last_fullsized":"2023-12-24T19:00:22.544205+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T01:58:57.653690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":284,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544525+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T19:00:22.544525+0000","last_peered":"2023-12-24T19:00:22.544525+0000","last_clean":"2023-12-24T19:00:22.544525+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T19:00:22.544525+0000","last_undegraded":"2023-12-24T19:00:22.544525+0000","last_fullsized":"2023-12-24T19:00:22.544525+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T22:45:34.173764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3529,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544114+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T19:00:22.544114+0000","last_peered":"2023-12-24T19:00:22.544114+0000","last_clean":"2023-12-24T19:00:22.544114+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T19:00:22.544114+0000","last_undegraded":"2023-12-24T19:00:22.544114+0000","last_fullsized":"2023-12-24T19:00:22.544114+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T23:52:54.059450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10731,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544559+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T19:00:22.544559+0000","last_peered":"2023-12-24T19:00:22.544559+0000","last_clean":"2023-12-24T19:00:22.544559+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T19:00:22.544559+0000","last_undegraded":"2023-12-24T19:00:22.544559+0000","last_fullsized":"2023-12-24T19:00:22.544559+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T19:02:38.676191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3502,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.543540+0000","last_change":"2023-12-24T18:02:38.129338+0000","last_active":"2023-12-24T19:00:22.543540+0000","last_peered":"2023-12-24T19:00:22.543540+0000","last_clean":"2023-12-24T19:00:22.543540+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T19:00:22.543540+0000","last_undegraded":"2023-12-24T19:00:22.543540+0000","last_fullsized":"2023-12-24T19:00:22.543540+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T22:55:41.056162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12783,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544395+0000","last_change":"2023-12-24T18:02:38.125565+0000","last_active":"2023-12-24T19:00:22.544395+0000","last_peered":"2023-12-24T19:00:22.544395+0000","last_clean":"2023-12-24T19:00:22.544395+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T19:00:22.544395+0000","last_undegraded":"2023-12-24T19:00:22.544395+0000","last_fullsized":"2023-12-24T19:00:22.544395+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T21:36:55.968107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3059,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544343+0000","last_change":"2023-12-24T18:02:38.124896+0000","last_active":"2023-12-24T19:00:22.544343+0000","last_peered":"2023-12-24T19:00:22.544343+0000","last_clean":"2023-12-24T19:00:22.544343+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T19:00:22.544343+0000","last_undegraded":"2023-12-24T19:00:22.544343+0000","last_fullsized":"2023-12-24T19:00:22.544343+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T00:37:18.060893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3313,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544290+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T19:00:22.544290+0000","last_peered":"2023-12-24T19:00:22.544290+0000","last_clean":"2023-12-24T19:00:22.544290+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T19:00:22.544290+0000","last_undegraded":"2023-12-24T19:00:22.544290+0000","last_fullsized":"2023-12-24T19:00:22.544290+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T01:45:50.432394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740619,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,2,6,12,4],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:03:49 2023","interfaces":[{"interface":"back","average":{"1min":0.20100000000000001,"5min":0.21199999999999999,"15min":0.21299999999999999},"min":{"1min":0.16200000000000001,"5min":0.155,"15min":0.14099999999999999},"max":{"1min":0.33700000000000002,"5min":0.40000000000000002,"15min":0.47499999999999998},"last":0.186},{"interface":"front","average":{"1min":0.23000000000000001,"5min":0.23400000000000001,"15min":0.23899999999999999},"min":{"1min":0.186,"5min":0.17199999999999999,"15min":0.16500000000000001},"max":{"1min":0.377,"5min":0.45100000000000001,"15min":0.48699999999999999},"last":0.21299999999999999}]},{"osd":1,"last update":"Sun Dec 24 21:03:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"min":{"1min":0.19,"5min":0.17299999999999999,"15min":0.16400000000000001},"max":{"1min":0.36299999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.26000000000000001,"5min":0.25600000000000001,"15min":0.25900000000000001},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.41199999999999998,"5min":0.45000000000000001,"15min":0.49299999999999999},"last":0.23999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359740618,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:03:57 2023","interfaces":[{"interface":"back","average":{"1min":0.183,"5min":0.20100000000000001,"15min":0.20200000000000001},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.127},"max":{"1min":0.35099999999999998,"5min":0.39400000000000002,"15min":0.44800000000000001},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.221,"5min":0.23000000000000001,"15min":0.22800000000000001},"min":{"1min":0.17299999999999999,"5min":0.155,"15min":0.14499999999999999},"max":{"1min":0.41799999999999998,"5min":0.45400000000000001,"15min":0.47599999999999998},"last":0.187}]},{"osd":2,"last update":"Sun Dec 24 21:03:57 2023","interfaces":[{"interface":"back","average":{"1min":0.25,"5min":0.252,"15min":0.247},"min":{"1min":0.20100000000000001,"5min":0.17899999999999999,"15min":0.158},"max":{"1min":0.38800000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.27800000000000002,"5min":0.27900000000000003,"15min":0.27600000000000002},"min":{"1min":0.22,"5min":0.188,"15min":0.188},"max":{"1min":0.45400000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.246}]}]},{"osd":1,"up_from":8,"seq":34359740621,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:04:02 2023","interfaces":[{"interface":"back","average":{"1min":0.188,"5min":0.19900000000000001,"15min":0.20000000000000001},"min":{"1min":0.14199999999999999,"5min":0.14199999999999999,"15min":0.13300000000000001},"max":{"1min":0.34300000000000003,"5min":0.38,"15min":0.40000000000000002},"last":0.16200000000000001},{"interface":"front","average":{"1min":0.222,"5min":0.22700000000000001,"15min":0.22700000000000001},"min":{"1min":0.17699999999999999,"5min":0.17000000000000001,"15min":0.159},"max":{"1min":0.373,"5min":0.41999999999999998,"15min":0.45800000000000002},"last":0.191}]},{"osd":2,"last update":"Sun Dec 24 21:04:02 2023","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":0.245,"15min":0.23999999999999999},"min":{"1min":0.16,"5min":0.16,"15min":0.16},"max":{"1min":0.432,"5min":0.46100000000000002,"15min":0.46999999999999997},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.26500000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"min":{"1min":0.19600000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.42399999999999999,"5min":0.45100000000000001,"15min":0.46800000000000003},"last":0.214}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:04:44.532 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:04:44.532 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:04:44.532 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:04:44.532 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:04:44.532 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:04:44.532 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:04:44.532 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:04:44.533 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:04:44.533 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:04:44.533 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:04:44.533 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:04:44.533 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:04:44.533 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:04:44.533 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:04:44.788 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:44.798+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:44.792 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:44.802+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:45.204 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:45.214+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:45.794 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:45.804+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:45.838 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:45.848+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:46.235 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:46.245+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:46.747 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:46.757+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:46.828 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:46.838+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:47.190 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:47.199+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:47.727 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:47.737+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:47.779 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:47.789+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:48.177 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:48.187+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:48.684 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:48.694+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:48.777 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:48.787+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:49.201 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:49.211+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:49.642 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:49.652+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:49.745 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:49.755+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:50.208 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:50.218+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:50.669 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:50.679+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:50.750 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:50.760+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:51.160 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:51.169+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:51.701 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:51.710+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:51.747 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:51.756+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:52.161 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:52.170+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:52.688 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:52.697+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:52.786 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:52.795+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:53.162 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:53.171+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:53.684 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:53.693+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:53.817 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:53.826+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:54.198 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:54.207+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:54.672 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:54.681+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:54.778 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:54.787+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:55.178 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:55.186+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:55.706 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:55.714+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:55.815 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:55.824+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:56.210 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:56.218+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:56.678 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:56.687+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:56.788 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:56.797+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:57.223 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:57.232+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:57.712 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:57.721+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:57.786 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:57.795+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:58.253 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:58.262+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:58.758 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:58.766+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:58.759 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:58.768+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:59.278 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:04:59.287+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:59.764 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:04:59.773+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:04:59.776 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:04:59.785+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:00.302 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:00.310+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:00.745 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:00.754+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:00.797 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:00.806+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:01.343 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:01.352+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:01.751 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:01.760+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:01.775 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:01.783+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:02.379 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:02.388+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:02.779 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:02.787+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:02.784 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:02.793+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:03.331 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:03.340+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:03.772 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:03.780+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:03.812 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:03.821+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:04.335 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:04.343+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:04.535 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:05:04.741 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:04.750+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:04.863 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:04.872+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:04.889 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:05:04.891 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:05:04.902 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":5781,"stamp":"2023-12-24T21:05:04.617199+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,2,6,12,516],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001741"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2824,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544349+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T19:00:22.544349+0000","last_peered":"2023-12-24T19:00:22.544349+0000","last_clean":"2023-12-24T19:00:22.544349+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T19:00:22.544349+0000","last_undegraded":"2023-12-24T19:00:22.544349+0000","last_fullsized":"2023-12-24T19:00:22.544349+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T20:39:55.726631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3618,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544409+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T19:00:22.544409+0000","last_peered":"2023-12-24T19:00:22.544409+0000","last_clean":"2023-12-24T19:00:22.544409+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T19:00:22.544409+0000","last_undegraded":"2023-12-24T19:00:22.544409+0000","last_fullsized":"2023-12-24T19:00:22.544409+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T01:55:17.265710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2981,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544459+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T19:00:22.544459+0000","last_peered":"2023-12-24T19:00:22.544459+0000","last_clean":"2023-12-24T19:00:22.544459+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T19:00:22.544459+0000","last_undegraded":"2023-12-24T19:00:22.544459+0000","last_fullsized":"2023-12-24T19:00:22.544459+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T00:30:36.804309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10075,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544271+0000","last_change":"2023-12-24T18:02:38.125769+0000","last_active":"2023-12-24T19:00:22.544271+0000","last_peered":"2023-12-24T19:00:22.544271+0000","last_clean":"2023-12-24T19:00:22.544271+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T19:00:22.544271+0000","last_undegraded":"2023-12-24T19:00:22.544271+0000","last_fullsized":"2023-12-24T19:00:22.544271+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T19:35:46.615178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3923,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.542749+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T19:00:52.542749+0000","last_peered":"2023-12-24T19:00:52.542749+0000","last_clean":"2023-12-24T19:00:52.542749+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T19:00:52.542749+0000","last_undegraded":"2023-12-24T19:00:52.542749+0000","last_fullsized":"2023-12-24T19:00:52.542749+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11203,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544205+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T19:00:22.544205+0000","last_peered":"2023-12-24T19:00:22.544205+0000","last_clean":"2023-12-24T19:00:22.544205+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T19:00:22.544205+0000","last_undegraded":"2023-12-24T19:00:22.544205+0000","last_fullsized":"2023-12-24T19:00:22.544205+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T01:58:57.653690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":284,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544525+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T19:00:22.544525+0000","last_peered":"2023-12-24T19:00:22.544525+0000","last_clean":"2023-12-24T19:00:22.544525+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T19:00:22.544525+0000","last_undegraded":"2023-12-24T19:00:22.544525+0000","last_fullsized":"2023-12-24T19:00:22.544525+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T22:45:34.173764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3529,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544114+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T19:00:22.544114+0000","last_peered":"2023-12-24T19:00:22.544114+0000","last_clean":"2023-12-24T19:00:22.544114+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T19:00:22.544114+0000","last_undegraded":"2023-12-24T19:00:22.544114+0000","last_fullsized":"2023-12-24T19:00:22.544114+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T23:52:54.059450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10731,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544559+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T19:00:22.544559+0000","last_peered":"2023-12-24T19:00:22.544559+0000","last_clean":"2023-12-24T19:00:22.544559+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T19:00:22.544559+0000","last_undegraded":"2023-12-24T19:00:22.544559+0000","last_fullsized":"2023-12-24T19:00:22.544559+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T19:02:38.676191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3507,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:04:46.246456+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:04:46.246456+0000","last_peered":"2023-12-24T21:04:46.246456+0000","last_clean":"2023-12-24T21:04:46.246456+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:04:46.246456+0000","last_undegraded":"2023-12-24T21:04:46.246456+0000","last_fullsized":"2023-12-24T21:04:46.246456+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12783,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544395+0000","last_change":"2023-12-24T18:02:38.125565+0000","last_active":"2023-12-24T19:00:22.544395+0000","last_peered":"2023-12-24T19:00:22.544395+0000","last_clean":"2023-12-24T19:00:22.544395+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T19:00:22.544395+0000","last_undegraded":"2023-12-24T19:00:22.544395+0000","last_fullsized":"2023-12-24T19:00:22.544395+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T21:36:55.968107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3059,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544343+0000","last_change":"2023-12-24T18:02:38.124896+0000","last_active":"2023-12-24T19:00:22.544343+0000","last_peered":"2023-12-24T19:00:22.544343+0000","last_clean":"2023-12-24T19:00:22.544343+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T19:00:22.544343+0000","last_undegraded":"2023-12-24T19:00:22.544343+0000","last_fullsized":"2023-12-24T19:00:22.544343+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T00:37:18.060893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3313,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544290+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T19:00:22.544290+0000","last_peered":"2023-12-24T19:00:22.544290+0000","last_clean":"2023-12-24T19:00:22.544290+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T19:00:22.544290+0000","last_undegraded":"2023-12-24T19:00:22.544290+0000","last_fullsized":"2023-12-24T19:00:22.544290+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T01:45:50.432394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740623,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,2,6,12,4],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:04:52 2023","interfaces":[{"interface":"back","average":{"1min":0.218,"5min":0.215,"15min":0.214},"min":{"1min":0.16,"5min":0.156,"15min":0.14099999999999999},"max":{"1min":0.38800000000000001,"5min":0.40000000000000002,"15min":0.47499999999999998},"last":0.20799999999999999},{"interface":"front","average":{"1min":0.25900000000000001,"5min":0.24099999999999999,"15min":0.23999999999999999},"min":{"1min":0.19500000000000001,"5min":0.17499999999999999,"15min":0.16500000000000001},"max":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.48699999999999999},"last":0.182}]},{"osd":1,"last update":"Sun Dec 24 21:04:52 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.23899999999999999,"15min":0.23400000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.16400000000000001},"max":{"1min":0.41499999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.26900000000000002,"5min":0.26200000000000001,"15min":0.26000000000000001},"min":{"1min":0.20799999999999999,"5min":0.182,"15min":0.17799999999999999},"max":{"1min":0.44,"5min":0.45000000000000001,"15min":0.49299999999999999},"last":0.222}]}]},{"osd":0,"up_from":8,"seq":34359740622,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:04:59 2023","interfaces":[{"interface":"back","average":{"1min":0.189,"5min":0.19900000000000001,"15min":0.20200000000000001},"min":{"1min":0.14999999999999999,"5min":0.14299999999999999,"15min":0.127},"max":{"1min":0.26500000000000001,"5min":0.39400000000000002,"15min":0.44800000000000001},"last":0.157},{"interface":"front","average":{"1min":0.222,"5min":0.23000000000000001,"15min":0.22800000000000001},"min":{"1min":0.17399999999999999,"5min":0.155,"15min":0.14499999999999999},"max":{"1min":0.312,"5min":0.45400000000000001,"15min":0.47599999999999998},"last":0.20300000000000001}]},{"osd":2,"last update":"Sun Dec 24 21:04:59 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.252,"15min":0.248},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.16800000000000001},"max":{"1min":0.34899999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.26800000000000002,"5min":0.27900000000000003,"15min":0.27700000000000002},"min":{"1min":0.20000000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.373,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.22500000000000001}]}]},{"osd":1,"up_from":8,"seq":34359740625,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:04:02 2023","interfaces":[{"interface":"back","average":{"1min":0.188,"5min":0.19900000000000001,"15min":0.20000000000000001},"min":{"1min":0.14199999999999999,"5min":0.14199999999999999,"15min":0.13300000000000001},"max":{"1min":0.34300000000000003,"5min":0.38,"15min":0.40000000000000002},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.222,"5min":0.22700000000000001,"15min":0.22700000000000001},"min":{"1min":0.17699999999999999,"5min":0.17000000000000001,"15min":0.159},"max":{"1min":0.373,"5min":0.41999999999999998,"15min":0.45800000000000002},"last":0.42699999999999999}]},{"osd":2,"last update":"Sun Dec 24 21:04:02 2023","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":0.245,"15min":0.23999999999999999},"min":{"1min":0.16,"5min":0.16,"15min":0.16},"max":{"1min":0.432,"5min":0.46100000000000002,"15min":0.46999999999999997},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.26500000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"min":{"1min":0.19600000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.42399999999999999,"5min":0.45100000000000001,"15min":0.46800000000000003},"last":0.45500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:05:04.904 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:04.904 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:04.904 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:04.904 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:04.904 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:04.904 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:04.905 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:04.905 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:04.905 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:04.905 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:04.905 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:04.905 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:04.906 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:04.906 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:05:05.348 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:05.356+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:05.701 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:05.710+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:05.866 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:05.875+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:06.360 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:06.369+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:06.739 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:06.747+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:06.863 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:06.871+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:07.352 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:07.360+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:07.733 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:07.741+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:07.866 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:07.874+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:08.325 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:08.334+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:08.736 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:08.745+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:08.822 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:08.831+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:09.283 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:09.292+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:09.760 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:09.769+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:09.862 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:09.871+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:10.288 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:10.297+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:10.728 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:10.737+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:10.862 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:10.871+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:11.284 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:11.293+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:11.688 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:11.696+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:11.907 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:11.916+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:12.263 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:12.271+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:12.728 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:12.737+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:12.941 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:12.949+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:13.235 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:13.242+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:13.752 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:13.761+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:13.953 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:13.961+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:14.201 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:14.210+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:14.784 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:14.793+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:14.968 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:14.976+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:15.156 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:15.164+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:15.737 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:15.745+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:15.988 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:15.997+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:16.110 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:16.118+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:16.708 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:16.717+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:17.012 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:17.020+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:17.127 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:17.136+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:17.727 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:17.735+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:18.019 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:18.027+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:18.086 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:18.093+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:18.694 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:18.702+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:19.011 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:19.019+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:19.092 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:19.100+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:19.722 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:19.730+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:20.040 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:20.048+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:20.072 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:20.080+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:20.704 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:20.712+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:21.053 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:21.062+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:21.104 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:21.113+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:21.671 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:21.679+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:22.103 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:22.112+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:22.155 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:22.164+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:22.669 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:22.677+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:23.108 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:23.118+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:23.124 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:23.134+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:23.649 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:23.659+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:24.106 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:24.115+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:24.109 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:24.119+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:24.679 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:24.688+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:24.907 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:05:25.074 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:25.083+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:25.091 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:25.100+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:25.246 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:05:25.249 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:05:25.260 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":5791,"stamp":"2023-12-24T21:05:24.620423+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,2,6,12,516],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002151"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2824,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544349+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T19:00:22.544349+0000","last_peered":"2023-12-24T19:00:22.544349+0000","last_clean":"2023-12-24T19:00:22.544349+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T19:00:22.544349+0000","last_undegraded":"2023-12-24T19:00:22.544349+0000","last_fullsized":"2023-12-24T19:00:22.544349+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T20:39:55.726631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3618,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544409+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T19:00:22.544409+0000","last_peered":"2023-12-24T19:00:22.544409+0000","last_clean":"2023-12-24T19:00:22.544409+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T19:00:22.544409+0000","last_undegraded":"2023-12-24T19:00:22.544409+0000","last_fullsized":"2023-12-24T19:00:22.544409+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T01:55:17.265710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2981,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544459+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T19:00:22.544459+0000","last_peered":"2023-12-24T19:00:22.544459+0000","last_clean":"2023-12-24T19:00:22.544459+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T19:00:22.544459+0000","last_undegraded":"2023-12-24T19:00:22.544459+0000","last_fullsized":"2023-12-24T19:00:22.544459+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T00:30:36.804309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10075,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544271+0000","last_change":"2023-12-24T18:02:38.125769+0000","last_active":"2023-12-24T19:00:22.544271+0000","last_peered":"2023-12-24T19:00:22.544271+0000","last_clean":"2023-12-24T19:00:22.544271+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T19:00:22.544271+0000","last_undegraded":"2023-12-24T19:00:22.544271+0000","last_fullsized":"2023-12-24T19:00:22.544271+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T19:35:46.615178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3923,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.542749+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T19:00:52.542749+0000","last_peered":"2023-12-24T19:00:52.542749+0000","last_clean":"2023-12-24T19:00:52.542749+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T19:00:52.542749+0000","last_undegraded":"2023-12-24T19:00:52.542749+0000","last_fullsized":"2023-12-24T19:00:52.542749+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11203,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544205+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T19:00:22.544205+0000","last_peered":"2023-12-24T19:00:22.544205+0000","last_clean":"2023-12-24T19:00:22.544205+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T19:00:22.544205+0000","last_undegraded":"2023-12-24T19:00:22.544205+0000","last_fullsized":"2023-12-24T19:00:22.544205+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T01:58:57.653690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":284,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544525+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T19:00:22.544525+0000","last_peered":"2023-12-24T19:00:22.544525+0000","last_clean":"2023-12-24T19:00:22.544525+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T19:00:22.544525+0000","last_undegraded":"2023-12-24T19:00:22.544525+0000","last_fullsized":"2023-12-24T19:00:22.544525+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T22:45:34.173764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3529,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544114+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T19:00:22.544114+0000","last_peered":"2023-12-24T19:00:22.544114+0000","last_clean":"2023-12-24T19:00:22.544114+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T19:00:22.544114+0000","last_undegraded":"2023-12-24T19:00:22.544114+0000","last_fullsized":"2023-12-24T19:00:22.544114+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T23:52:54.059450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10731,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544559+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T19:00:22.544559+0000","last_peered":"2023-12-24T19:00:22.544559+0000","last_clean":"2023-12-24T19:00:22.544559+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T19:00:22.544559+0000","last_undegraded":"2023-12-24T19:00:22.544559+0000","last_fullsized":"2023-12-24T19:00:22.544559+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T19:02:38.676191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3507,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:04:46.246456+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:04:46.246456+0000","last_peered":"2023-12-24T21:04:46.246456+0000","last_clean":"2023-12-24T21:04:46.246456+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:04:46.246456+0000","last_undegraded":"2023-12-24T21:04:46.246456+0000","last_fullsized":"2023-12-24T21:04:46.246456+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12783,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544395+0000","last_change":"2023-12-24T18:02:38.125565+0000","last_active":"2023-12-24T19:00:22.544395+0000","last_peered":"2023-12-24T19:00:22.544395+0000","last_clean":"2023-12-24T19:00:22.544395+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T19:00:22.544395+0000","last_undegraded":"2023-12-24T19:00:22.544395+0000","last_fullsized":"2023-12-24T19:00:22.544395+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T21:36:55.968107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3059,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544343+0000","last_change":"2023-12-24T18:02:38.124896+0000","last_active":"2023-12-24T19:00:22.544343+0000","last_peered":"2023-12-24T19:00:22.544343+0000","last_clean":"2023-12-24T19:00:22.544343+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T19:00:22.544343+0000","last_undegraded":"2023-12-24T19:00:22.544343+0000","last_fullsized":"2023-12-24T19:00:22.544343+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T00:37:18.060893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3313,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544290+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T19:00:22.544290+0000","last_peered":"2023-12-24T19:00:22.544290+0000","last_clean":"2023-12-24T19:00:22.544290+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T19:00:22.544290+0000","last_undegraded":"2023-12-24T19:00:22.544290+0000","last_fullsized":"2023-12-24T19:00:22.544290+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T01:45:50.432394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740627,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,2,6,12,4],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:04:52 2023","interfaces":[{"interface":"back","average":{"1min":0.218,"5min":0.215,"15min":0.214},"min":{"1min":0.16,"5min":0.156,"15min":0.14099999999999999},"max":{"1min":0.38800000000000001,"5min":0.40000000000000002,"15min":0.47499999999999998},"last":0.16400000000000001},{"interface":"front","average":{"1min":0.25900000000000001,"5min":0.24099999999999999,"15min":0.23999999999999999},"min":{"1min":0.19500000000000001,"5min":0.17499999999999999,"15min":0.16500000000000001},"max":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.48699999999999999},"last":0.187}]},{"osd":1,"last update":"Sun Dec 24 21:04:52 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.23899999999999999,"15min":0.23400000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.16400000000000001},"max":{"1min":0.41499999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"last":0.20100000000000001},{"interface":"front","average":{"1min":0.26900000000000002,"5min":0.26200000000000001,"15min":0.26000000000000001},"min":{"1min":0.20799999999999999,"5min":0.182,"15min":0.17799999999999999},"max":{"1min":0.44,"5min":0.45000000000000001,"15min":0.49299999999999999},"last":0.223}]}]},{"osd":0,"up_from":8,"seq":34359740626,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:04:59 2023","interfaces":[{"interface":"back","average":{"1min":0.189,"5min":0.19900000000000001,"15min":0.20200000000000001},"min":{"1min":0.14999999999999999,"5min":0.14299999999999999,"15min":0.127},"max":{"1min":0.26500000000000001,"5min":0.39400000000000002,"15min":0.44800000000000001},"last":0.185},{"interface":"front","average":{"1min":0.222,"5min":0.23000000000000001,"15min":0.22800000000000001},"min":{"1min":0.17399999999999999,"5min":0.155,"15min":0.14499999999999999},"max":{"1min":0.312,"5min":0.45400000000000001,"15min":0.47599999999999998},"last":0.22600000000000001}]},{"osd":2,"last update":"Sun Dec 24 21:04:59 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.252,"15min":0.248},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.16800000000000001},"max":{"1min":0.34899999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.26800000000000002,"5min":0.27900000000000003,"15min":0.27700000000000002},"min":{"1min":0.20000000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.373,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.28699999999999998}]}]},{"osd":1,"up_from":8,"seq":34359740629,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:05:02 2023","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":0.20399999999999999,"15min":0.20100000000000001},"min":{"1min":0.13300000000000001,"5min":0.13300000000000001,"15min":0.13300000000000001},"max":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"last":0.16400000000000001},{"interface":"front","average":{"1min":0.248,"5min":0.23200000000000001,"15min":0.22900000000000001},"min":{"1min":0.161,"5min":0.161,"15min":0.159},"max":{"1min":0.439,"5min":0.439,"15min":0.45800000000000002},"last":0.20799999999999999}]},{"osd":2,"last update":"Sun Dec 24 21:05:02 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.252,"15min":0.24199999999999999},"min":{"1min":0.184,"5min":0.16,"15min":0.16},"max":{"1min":0.45600000000000002,"5min":0.45900000000000002,"15min":0.46999999999999997},"last":0.192},{"interface":"front","average":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26400000000000001},"min":{"1min":0.20300000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"last":0.25800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:05:25.261 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:25.261 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:25.262 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:25.262 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:25.262 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:25.262 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:25.262 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:25.262 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:25.263 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:25.263 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:25.263 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:25.263 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:25.263 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:25.263 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:05:25.667 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:25.677+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:26.056 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:26.065+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:26.087 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:26.096+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:26.632 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:26.641+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:27.094 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:27.103+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:27.108 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:27.117+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:27.588 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:27.597+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:28.082 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:28.091+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:28.091 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:28.101+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:28.555 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:28.564+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:29.097 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:29.106+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:29.120 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:29.129+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:29.578 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:29.587+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:30.073 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:30.081+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:30.123 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:30.132+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:30.627 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:30.636+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:31.089 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:31.098+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:31.103 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:31.113+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:31.612 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:31.621+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:32.068 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:32.077+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:32.070 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:32.079+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:32.622 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:32.631+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:33.044 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:33.053+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:33.115 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:33.125+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:33.670 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:33.679+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:34.084 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:34.093+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:34.089 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:34.099+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:34.674 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:34.683+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:35.100 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:35.109+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:35.120 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:35.129+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:35.701 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:35.710+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:36.134 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:36.142+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:36.169 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:36.178+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:36.704 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:36.712+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:37.162 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:37.170+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:37.203 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:37.211+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:37.708 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:37.716+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:38.169 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:38.177+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:38.184 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:38.192+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:38.716 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:38.724+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:39.167 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:39.175+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:39.194 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:39.202+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:39.674 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:39.682+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:40.171 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:40.179+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:40.192 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:40.200+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:40.712 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:40.720+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:41.164 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:41.172+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:41.202 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:41.210+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:41.718 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:41.726+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:42.164 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:42.171+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:42.202 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:42.210+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:42.672 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:42.679+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:43.168 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:43.176+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:43.199 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:43.207+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:43.655 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:43.663+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:44.167 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:44.175+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:44.189 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:44.197+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:44.683 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:44.691+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:45.166 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:45.174+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:45.223 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:45.230+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:45.264 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:05:45.585 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:05:45.587 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:05:45.597 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":5801,"stamp":"2023-12-24T21:05:44.623714+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,2,6,12,516],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001925"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2824,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544349+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T19:00:22.544349+0000","last_peered":"2023-12-24T19:00:22.544349+0000","last_clean":"2023-12-24T19:00:22.544349+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T19:00:22.544349+0000","last_undegraded":"2023-12-24T19:00:22.544349+0000","last_fullsized":"2023-12-24T19:00:22.544349+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T20:39:55.726631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3618,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544409+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T19:00:22.544409+0000","last_peered":"2023-12-24T19:00:22.544409+0000","last_clean":"2023-12-24T19:00:22.544409+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T19:00:22.544409+0000","last_undegraded":"2023-12-24T19:00:22.544409+0000","last_fullsized":"2023-12-24T19:00:22.544409+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T01:55:17.265710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2981,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544459+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T19:00:22.544459+0000","last_peered":"2023-12-24T19:00:22.544459+0000","last_clean":"2023-12-24T19:00:22.544459+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T19:00:22.544459+0000","last_undegraded":"2023-12-24T19:00:22.544459+0000","last_fullsized":"2023-12-24T19:00:22.544459+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T00:30:36.804309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10075,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544271+0000","last_change":"2023-12-24T18:02:38.125769+0000","last_active":"2023-12-24T19:00:22.544271+0000","last_peered":"2023-12-24T19:00:22.544271+0000","last_clean":"2023-12-24T19:00:22.544271+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T19:00:22.544271+0000","last_undegraded":"2023-12-24T19:00:22.544271+0000","last_fullsized":"2023-12-24T19:00:22.544271+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T19:35:46.615178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3923,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.542749+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T19:00:52.542749+0000","last_peered":"2023-12-24T19:00:52.542749+0000","last_clean":"2023-12-24T19:00:52.542749+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T19:00:52.542749+0000","last_undegraded":"2023-12-24T19:00:52.542749+0000","last_fullsized":"2023-12-24T19:00:52.542749+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11203,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544205+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T19:00:22.544205+0000","last_peered":"2023-12-24T19:00:22.544205+0000","last_clean":"2023-12-24T19:00:22.544205+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T19:00:22.544205+0000","last_undegraded":"2023-12-24T19:00:22.544205+0000","last_fullsized":"2023-12-24T19:00:22.544205+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T01:58:57.653690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":284,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544525+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T19:00:22.544525+0000","last_peered":"2023-12-24T19:00:22.544525+0000","last_clean":"2023-12-24T19:00:22.544525+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T19:00:22.544525+0000","last_undegraded":"2023-12-24T19:00:22.544525+0000","last_fullsized":"2023-12-24T19:00:22.544525+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T22:45:34.173764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3529,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544114+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T19:00:22.544114+0000","last_peered":"2023-12-24T19:00:22.544114+0000","last_clean":"2023-12-24T19:00:22.544114+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T19:00:22.544114+0000","last_undegraded":"2023-12-24T19:00:22.544114+0000","last_fullsized":"2023-12-24T19:00:22.544114+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T23:52:54.059450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10731,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544559+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T19:00:22.544559+0000","last_peered":"2023-12-24T19:00:22.544559+0000","last_clean":"2023-12-24T19:00:22.544559+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T19:00:22.544559+0000","last_undegraded":"2023-12-24T19:00:22.544559+0000","last_fullsized":"2023-12-24T19:00:22.544559+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T19:02:38.676191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3507,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:04:46.246456+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:04:46.246456+0000","last_peered":"2023-12-24T21:04:46.246456+0000","last_clean":"2023-12-24T21:04:46.246456+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:04:46.246456+0000","last_undegraded":"2023-12-24T21:04:46.246456+0000","last_fullsized":"2023-12-24T21:04:46.246456+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12783,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544395+0000","last_change":"2023-12-24T18:02:38.125565+0000","last_active":"2023-12-24T19:00:22.544395+0000","last_peered":"2023-12-24T19:00:22.544395+0000","last_clean":"2023-12-24T19:00:22.544395+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T19:00:22.544395+0000","last_undegraded":"2023-12-24T19:00:22.544395+0000","last_fullsized":"2023-12-24T19:00:22.544395+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T21:36:55.968107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3059,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544343+0000","last_change":"2023-12-24T18:02:38.124896+0000","last_active":"2023-12-24T19:00:22.544343+0000","last_peered":"2023-12-24T19:00:22.544343+0000","last_clean":"2023-12-24T19:00:22.544343+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T19:00:22.544343+0000","last_undegraded":"2023-12-24T19:00:22.544343+0000","last_fullsized":"2023-12-24T19:00:22.544343+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T00:37:18.060893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3313,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544290+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T19:00:22.544290+0000","last_peered":"2023-12-24T19:00:22.544290+0000","last_clean":"2023-12-24T19:00:22.544290+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T19:00:22.544290+0000","last_undegraded":"2023-12-24T19:00:22.544290+0000","last_fullsized":"2023-12-24T19:00:22.544290+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T01:45:50.432394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740631,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,2,6,12,4],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:04:52 2023","interfaces":[{"interface":"back","average":{"1min":0.218,"5min":0.215,"15min":0.214},"min":{"1min":0.16,"5min":0.156,"15min":0.14099999999999999},"max":{"1min":0.38800000000000001,"5min":0.40000000000000002,"15min":0.47499999999999998},"last":0.151},{"interface":"front","average":{"1min":0.25900000000000001,"5min":0.24099999999999999,"15min":0.23999999999999999},"min":{"1min":0.19500000000000001,"5min":0.17499999999999999,"15min":0.16500000000000001},"max":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.48699999999999999},"last":0.189}]},{"osd":1,"last update":"Sun Dec 24 21:04:52 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.23899999999999999,"15min":0.23400000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.16400000000000001},"max":{"1min":0.41499999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.26900000000000002,"5min":0.26200000000000001,"15min":0.26000000000000001},"min":{"1min":0.20799999999999999,"5min":0.182,"15min":0.17799999999999999},"max":{"1min":0.44,"5min":0.45000000000000001,"15min":0.49299999999999999},"last":0.24299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359740630,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:04:59 2023","interfaces":[{"interface":"back","average":{"1min":0.189,"5min":0.19900000000000001,"15min":0.20200000000000001},"min":{"1min":0.14999999999999999,"5min":0.14299999999999999,"15min":0.127},"max":{"1min":0.26500000000000001,"5min":0.39400000000000002,"15min":0.44800000000000001},"last":0.16600000000000001},{"interface":"front","average":{"1min":0.222,"5min":0.23000000000000001,"15min":0.22800000000000001},"min":{"1min":0.17399999999999999,"5min":0.155,"15min":0.14499999999999999},"max":{"1min":0.312,"5min":0.45400000000000001,"15min":0.47599999999999998},"last":0.19500000000000001}]},{"osd":2,"last update":"Sun Dec 24 21:04:59 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.252,"15min":0.248},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.16800000000000001},"max":{"1min":0.34899999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.26800000000000002,"5min":0.27900000000000003,"15min":0.27700000000000002},"min":{"1min":0.20000000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.373,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.32800000000000001}]}]},{"osd":1,"up_from":8,"seq":34359740633,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:05:02 2023","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":0.20399999999999999,"15min":0.20100000000000001},"min":{"1min":0.13300000000000001,"5min":0.13300000000000001,"15min":0.13300000000000001},"max":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"last":0.17499999999999999},{"interface":"front","average":{"1min":0.248,"5min":0.23200000000000001,"15min":0.22900000000000001},"min":{"1min":0.161,"5min":0.161,"15min":0.159},"max":{"1min":0.439,"5min":0.439,"15min":0.45800000000000002},"last":0.21099999999999999}]},{"osd":2,"last update":"Sun Dec 24 21:05:02 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.252,"15min":0.24199999999999999},"min":{"1min":0.184,"5min":0.16,"15min":0.16},"max":{"1min":0.45600000000000002,"5min":0.45900000000000002,"15min":0.46999999999999997},"last":0.191},{"interface":"front","average":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26400000000000001},"min":{"1min":0.20300000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"last":0.24299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:05:45.599 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:45.599 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:45.599 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:45.599 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:45.599 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:45.600 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:45.600 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:45.600 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:45.600 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:45.600 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:45.601 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:45.601 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:45.601 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:05:45.601 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:05:45.639 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:45.647+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:46.196 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:46.203+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:46.264 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:46.272+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:46.648 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:46.656+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:47.176 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:47.184+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:47.287 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:47.294+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:47.612 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:47.620+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:48.215 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:48.223+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:48.312 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:48.320+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:48.658 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:48.666+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:49.220 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:49.228+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:49.264 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:49.272+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:49.612 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:49.620+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:50.215 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:50.223+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:50.309 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:50.317+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:50.638 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:50.646+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:51.179 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:51.187+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:51.276 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:51.284+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:51.660 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:51.668+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:52.150 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:52.158+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:52.231 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:52.239+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:52.616 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:52.624+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:53.189 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:53.196+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:53.251 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:53.259+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:53.642 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:53.649+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:54.194 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:54.202+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:54.292 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:54.300+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:54.672 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:54.680+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:55.231 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:55.239+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:55.255 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:55.262+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:55.638 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:55.645+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:56.207 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:56.215+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:56.237 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:56.244+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:56.590 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:56.598+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:57.204 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:57.212+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:57.257 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:57.265+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:57.596 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:57.604+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:58.201 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:58.209+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:58.263 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:58.271+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:58.597 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:58.605+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:59.238 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:05:59.245+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:59.290 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:05:59.297+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:05:59.634 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:05:59.642+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:00.231 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:00.238+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:00.259 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:00.267+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:00.679 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:00.686+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:01.182 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:01.190+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:01.212 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:01.219+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:01.698 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:01.705+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:02.208 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:02.216+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:02.216 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:02.224+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:02.742 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:02.750+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:03.221 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:03.228+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:03.246 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:03.254+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:03.696 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:03.703+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:04.226 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:04.233+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:04.271 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:04.279+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:04.706 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:04.714+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:05.200 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:05.207+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:05.293 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:05.300+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:05.603 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:06:05.716 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:05.724+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:05.960 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:06:05.963 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:06:05.974 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":5811,"stamp":"2023-12-24T21:06:04.626855+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,2,6,12,516],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001925"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2824,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544349+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T19:00:22.544349+0000","last_peered":"2023-12-24T19:00:22.544349+0000","last_clean":"2023-12-24T19:00:22.544349+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T19:00:22.544349+0000","last_undegraded":"2023-12-24T19:00:22.544349+0000","last_fullsized":"2023-12-24T19:00:22.544349+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T20:39:55.726631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3618,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544409+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T19:00:22.544409+0000","last_peered":"2023-12-24T19:00:22.544409+0000","last_clean":"2023-12-24T19:00:22.544409+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T19:00:22.544409+0000","last_undegraded":"2023-12-24T19:00:22.544409+0000","last_fullsized":"2023-12-24T19:00:22.544409+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T01:55:17.265710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2981,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544459+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T19:00:22.544459+0000","last_peered":"2023-12-24T19:00:22.544459+0000","last_clean":"2023-12-24T19:00:22.544459+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T19:00:22.544459+0000","last_undegraded":"2023-12-24T19:00:22.544459+0000","last_fullsized":"2023-12-24T19:00:22.544459+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T00:30:36.804309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10075,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544271+0000","last_change":"2023-12-24T18:02:38.125769+0000","last_active":"2023-12-24T19:00:22.544271+0000","last_peered":"2023-12-24T19:00:22.544271+0000","last_clean":"2023-12-24T19:00:22.544271+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T19:00:22.544271+0000","last_undegraded":"2023-12-24T19:00:22.544271+0000","last_fullsized":"2023-12-24T19:00:22.544271+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T19:35:46.615178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3923,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.542749+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T19:00:52.542749+0000","last_peered":"2023-12-24T19:00:52.542749+0000","last_clean":"2023-12-24T19:00:52.542749+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T19:00:52.542749+0000","last_undegraded":"2023-12-24T19:00:52.542749+0000","last_fullsized":"2023-12-24T19:00:52.542749+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11203,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544205+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T19:00:22.544205+0000","last_peered":"2023-12-24T19:00:22.544205+0000","last_clean":"2023-12-24T19:00:22.544205+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T19:00:22.544205+0000","last_undegraded":"2023-12-24T19:00:22.544205+0000","last_fullsized":"2023-12-24T19:00:22.544205+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T01:58:57.653690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":284,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544525+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T19:00:22.544525+0000","last_peered":"2023-12-24T19:00:22.544525+0000","last_clean":"2023-12-24T19:00:22.544525+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T19:00:22.544525+0000","last_undegraded":"2023-12-24T19:00:22.544525+0000","last_fullsized":"2023-12-24T19:00:22.544525+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T22:45:34.173764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3529,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544114+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T19:00:22.544114+0000","last_peered":"2023-12-24T19:00:22.544114+0000","last_clean":"2023-12-24T19:00:22.544114+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T19:00:22.544114+0000","last_undegraded":"2023-12-24T19:00:22.544114+0000","last_fullsized":"2023-12-24T19:00:22.544114+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T23:52:54.059450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10731,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544559+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T19:00:22.544559+0000","last_peered":"2023-12-24T19:00:22.544559+0000","last_clean":"2023-12-24T19:00:22.544559+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T19:00:22.544559+0000","last_undegraded":"2023-12-24T19:00:22.544559+0000","last_fullsized":"2023-12-24T19:00:22.544559+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T19:02:38.676191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3507,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:04:46.246456+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:04:46.246456+0000","last_peered":"2023-12-24T21:04:46.246456+0000","last_clean":"2023-12-24T21:04:46.246456+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:04:46.246456+0000","last_undegraded":"2023-12-24T21:04:46.246456+0000","last_fullsized":"2023-12-24T21:04:46.246456+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12783,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544395+0000","last_change":"2023-12-24T18:02:38.125565+0000","last_active":"2023-12-24T19:00:22.544395+0000","last_peered":"2023-12-24T19:00:22.544395+0000","last_clean":"2023-12-24T19:00:22.544395+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T19:00:22.544395+0000","last_undegraded":"2023-12-24T19:00:22.544395+0000","last_fullsized":"2023-12-24T19:00:22.544395+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T21:36:55.968107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3059,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544343+0000","last_change":"2023-12-24T18:02:38.124896+0000","last_active":"2023-12-24T19:00:22.544343+0000","last_peered":"2023-12-24T19:00:22.544343+0000","last_clean":"2023-12-24T19:00:22.544343+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T19:00:22.544343+0000","last_undegraded":"2023-12-24T19:00:22.544343+0000","last_fullsized":"2023-12-24T19:00:22.544343+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T00:37:18.060893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3313,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544290+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T19:00:22.544290+0000","last_peered":"2023-12-24T19:00:22.544290+0000","last_clean":"2023-12-24T19:00:22.544290+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T19:00:22.544290+0000","last_undegraded":"2023-12-24T19:00:22.544290+0000","last_fullsized":"2023-12-24T19:00:22.544290+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T01:45:50.432394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740635,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,2,6,12,4],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:05:56 2023","interfaces":[{"interface":"back","average":{"1min":0.216,"5min":0.21299999999999999,"15min":0.215},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14099999999999999},"max":{"1min":0.38900000000000001,"5min":0.40000000000000002,"15min":0.47499999999999998},"last":0.17499999999999999},{"interface":"front","average":{"1min":0.23799999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.16500000000000001},"max":{"1min":0.42999999999999999,"5min":0.45600000000000002,"15min":0.48699999999999999},"last":0.20200000000000001}]},{"osd":1,"last update":"Sun Dec 24 21:05:56 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.24099999999999999,"15min":0.23599999999999999},"min":{"1min":0.19,"5min":0.182,"15min":0.16400000000000001},"max":{"1min":0.44900000000000001,"5min":0.46300000000000002,"15min":0.46300000000000002},"last":0.219},{"interface":"front","average":{"1min":0.26800000000000002,"5min":0.26300000000000001,"15min":0.26200000000000001},"min":{"1min":0.20799999999999999,"5min":0.182,"15min":0.17799999999999999},"max":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.49299999999999999},"last":0.24199999999999999}]}]},{"osd":0,"up_from":8,"seq":34359740634,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:04:59 2023","interfaces":[{"interface":"back","average":{"1min":0.189,"5min":0.19900000000000001,"15min":0.20200000000000001},"min":{"1min":0.14999999999999999,"5min":0.14299999999999999,"15min":0.127},"max":{"1min":0.26500000000000001,"5min":0.39400000000000002,"15min":0.44800000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.222,"5min":0.23000000000000001,"15min":0.22800000000000001},"min":{"1min":0.17399999999999999,"5min":0.155,"15min":0.14499999999999999},"max":{"1min":0.312,"5min":0.45400000000000001,"15min":0.47599999999999998},"last":0.47199999999999998}]},{"osd":2,"last update":"Sun Dec 24 21:04:59 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.252,"15min":0.248},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.16800000000000001},"max":{"1min":0.34899999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"last":0.438},{"interface":"front","average":{"1min":0.26800000000000002,"5min":0.27900000000000003,"15min":0.27700000000000002},"min":{"1min":0.20000000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.373,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.495}]}]},{"osd":1,"up_from":8,"seq":34359740637,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:05:02 2023","interfaces":[{"interface":"back","average":{"1min":0.214,"5min":0.20399999999999999,"15min":0.20100000000000001},"min":{"1min":0.13300000000000001,"5min":0.13300000000000001,"15min":0.13300000000000001},"max":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"last":0.14599999999999999},{"interface":"front","average":{"1min":0.248,"5min":0.23200000000000001,"15min":0.22900000000000001},"min":{"1min":0.161,"5min":0.161,"15min":0.159},"max":{"1min":0.439,"5min":0.439,"15min":0.45800000000000002},"last":0.17199999999999999}]},{"osd":2,"last update":"Sun Dec 24 21:05:02 2023","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.252,"15min":0.24199999999999999},"min":{"1min":0.184,"5min":0.16,"15min":0.16},"max":{"1min":0.45600000000000002,"5min":0.45900000000000002,"15min":0.46999999999999997},"last":0.157},{"interface":"front","average":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26400000000000001},"min":{"1min":0.20300000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"last":0.20499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:06:05.975 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:05.975 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:05.976 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:05.976 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:05.976 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:05.976 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:05.976 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:05.976 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:05.976 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:05.976 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:05.977 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:05.977 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:05.977 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:05.977 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:06:06.179 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:06.187+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:06.342 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:06.350+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:06.742 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:06.749+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:07.220 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:07.225+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:07.305 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:07.312+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:07.789 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:07.798+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:08.246 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:08.255+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:08.344 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:08.352+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:08.777 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:08.785+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:09.250 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:09.258+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:09.372 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:09.380+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:09.729 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:09.737+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:10.270 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:10.278+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:10.407 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:10.416+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:10.684 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:10.692+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:11.275 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:11.283+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:11.372 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:11.381+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:11.690 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:11.698+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:12.310 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:12.319+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:12.422 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:12.431+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:12.647 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:12.655+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:13.289 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:13.297+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:13.411 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:13.419+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:13.692 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:13.700+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:14.288 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:14.296+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:14.412 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:14.420+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:14.691 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:14.699+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:15.309 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:15.317+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:15.375 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:15.384+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:15.700 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:15.708+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:16.348 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:16.356+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:16.359 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:16.367+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:16.654 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:16.663+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:17.320 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:17.328+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:17.358 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:17.367+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:17.691 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:17.700+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:18.295 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:18.304+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:18.369 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:18.378+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:18.697 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:18.705+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:19.287 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:19.295+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:19.412 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:19.420+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:19.677 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:19.685+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:20.320 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:20.328+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:20.390 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:20.399+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:20.664 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:20.671+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:21.280 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:21.287+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:21.410 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:21.417+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:21.641 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:21.648+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:22.260 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:22.267+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:22.418 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:22.425+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:22.638 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:22.645+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:23.272 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:23.279+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:23.427 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:23.434+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:23.639 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:23.646+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:24.230 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:24.237+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:24.457 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:24.464+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:24.632 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:24.639+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:25.228 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:25.235+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:25.496 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:25.503+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:25.655 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:25.662+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:25.978 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:06:26.214 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:26.221+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:26.341 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:06:26.343 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:06:26.354 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":5821,"stamp":"2023-12-24T21:06:24.629651+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,2,6,12,516],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001650"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2824,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544349+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T19:00:22.544349+0000","last_peered":"2023-12-24T19:00:22.544349+0000","last_clean":"2023-12-24T19:00:22.544349+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T19:00:22.544349+0000","last_undegraded":"2023-12-24T19:00:22.544349+0000","last_fullsized":"2023-12-24T19:00:22.544349+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T20:39:55.726631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3618,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544409+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T19:00:22.544409+0000","last_peered":"2023-12-24T19:00:22.544409+0000","last_clean":"2023-12-24T19:00:22.544409+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T19:00:22.544409+0000","last_undegraded":"2023-12-24T19:00:22.544409+0000","last_fullsized":"2023-12-24T19:00:22.544409+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T01:55:17.265710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2981,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544459+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T19:00:22.544459+0000","last_peered":"2023-12-24T19:00:22.544459+0000","last_clean":"2023-12-24T19:00:22.544459+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T19:00:22.544459+0000","last_undegraded":"2023-12-24T19:00:22.544459+0000","last_fullsized":"2023-12-24T19:00:22.544459+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T00:30:36.804309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10075,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544271+0000","last_change":"2023-12-24T18:02:38.125769+0000","last_active":"2023-12-24T19:00:22.544271+0000","last_peered":"2023-12-24T19:00:22.544271+0000","last_clean":"2023-12-24T19:00:22.544271+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T19:00:22.544271+0000","last_undegraded":"2023-12-24T19:00:22.544271+0000","last_fullsized":"2023-12-24T19:00:22.544271+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T19:35:46.615178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3923,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.542749+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T19:00:52.542749+0000","last_peered":"2023-12-24T19:00:52.542749+0000","last_clean":"2023-12-24T19:00:52.542749+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T19:00:52.542749+0000","last_undegraded":"2023-12-24T19:00:52.542749+0000","last_fullsized":"2023-12-24T19:00:52.542749+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11203,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544205+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T19:00:22.544205+0000","last_peered":"2023-12-24T19:00:22.544205+0000","last_clean":"2023-12-24T19:00:22.544205+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T19:00:22.544205+0000","last_undegraded":"2023-12-24T19:00:22.544205+0000","last_fullsized":"2023-12-24T19:00:22.544205+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T01:58:57.653690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":284,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544525+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T19:00:22.544525+0000","last_peered":"2023-12-24T19:00:22.544525+0000","last_clean":"2023-12-24T19:00:22.544525+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T19:00:22.544525+0000","last_undegraded":"2023-12-24T19:00:22.544525+0000","last_fullsized":"2023-12-24T19:00:22.544525+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T22:45:34.173764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3529,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544114+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T19:00:22.544114+0000","last_peered":"2023-12-24T19:00:22.544114+0000","last_clean":"2023-12-24T19:00:22.544114+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T19:00:22.544114+0000","last_undegraded":"2023-12-24T19:00:22.544114+0000","last_fullsized":"2023-12-24T19:00:22.544114+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T23:52:54.059450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10731,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544559+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T19:00:22.544559+0000","last_peered":"2023-12-24T19:00:22.544559+0000","last_clean":"2023-12-24T19:00:22.544559+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T19:00:22.544559+0000","last_undegraded":"2023-12-24T19:00:22.544559+0000","last_fullsized":"2023-12-24T19:00:22.544559+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T19:02:38.676191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3507,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:04:46.246456+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:04:46.246456+0000","last_peered":"2023-12-24T21:04:46.246456+0000","last_clean":"2023-12-24T21:04:46.246456+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:04:46.246456+0000","last_undegraded":"2023-12-24T21:04:46.246456+0000","last_fullsized":"2023-12-24T21:04:46.246456+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12783,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544395+0000","last_change":"2023-12-24T18:02:38.125565+0000","last_active":"2023-12-24T19:00:22.544395+0000","last_peered":"2023-12-24T19:00:22.544395+0000","last_clean":"2023-12-24T19:00:22.544395+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T19:00:22.544395+0000","last_undegraded":"2023-12-24T19:00:22.544395+0000","last_fullsized":"2023-12-24T19:00:22.544395+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-25T21:36:55.968107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3059,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544343+0000","last_change":"2023-12-24T18:02:38.124896+0000","last_active":"2023-12-24T19:00:22.544343+0000","last_peered":"2023-12-24T19:00:22.544343+0000","last_clean":"2023-12-24T19:00:22.544343+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T19:00:22.544343+0000","last_undegraded":"2023-12-24T19:00:22.544343+0000","last_fullsized":"2023-12-24T19:00:22.544343+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T00:37:18.060893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3313,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-12-24T19:00:22.544290+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T19:00:22.544290+0000","last_peered":"2023-12-24T19:00:22.544290+0000","last_clean":"2023-12-24T19:00:22.544290+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T19:00:22.544290+0000","last_undegraded":"2023-12-24T19:00:22.544290+0000","last_fullsized":"2023-12-24T19:00:22.544290+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-26T01:45:50.432394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740639,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,2,6,12,4],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:05:56 2023","interfaces":[{"interface":"back","average":{"1min":0.216,"5min":0.21299999999999999,"15min":0.215},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14099999999999999},"max":{"1min":0.38900000000000001,"5min":0.40000000000000002,"15min":0.47499999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.23799999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.16500000000000001},"max":{"1min":0.42999999999999999,"5min":0.45600000000000002,"15min":0.48699999999999999},"last":0.184}]},{"osd":1,"last update":"Sun Dec 24 21:05:56 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.24099999999999999,"15min":0.23599999999999999},"min":{"1min":0.19,"5min":0.182,"15min":0.16400000000000001},"max":{"1min":0.44900000000000001,"5min":0.46300000000000002,"15min":0.46300000000000002},"last":0.20200000000000001},{"interface":"front","average":{"1min":0.26800000000000002,"5min":0.26300000000000001,"15min":0.26200000000000001},"min":{"1min":0.20799999999999999,"5min":0.182,"15min":0.17799999999999999},"max":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.49299999999999999},"last":0.23300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359740638,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":0.19600000000000001,"15min":0.20100000000000001},"min":{"1min":0.155,"5min":0.14299999999999999,"15min":0.127},"max":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.44800000000000001},"last":0.183},{"interface":"front","average":{"1min":0.23300000000000001,"5min":0.22700000000000001,"15min":0.22900000000000001},"min":{"1min":0.183,"5min":0.155,"15min":0.14499999999999999},"max":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47599999999999998},"last":0.223}]},{"osd":2,"last update":"Sun Dec 24 21:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":0.249,"15min":0.249},"min":{"1min":0.186,"5min":0.17100000000000001,"15min":0.16800000000000001},"max":{"1min":0.438,"5min":0.48799999999999999,"15min":0.48799999999999999},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.28699999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"min":{"1min":0.23400000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.495,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.29199999999999998}]}]},{"osd":1,"up_from":8,"seq":34359740641,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.18099999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"min":{"1min":0.14599999999999999,"5min":0.13300000000000001,"15min":0.13300000000000001},"max":{"1min":0.437,"5min":0.437,"15min":0.437},"last":0.183},{"interface":"front","average":{"1min":0.20799999999999999,"5min":0.22700000000000001,"15min":0.22800000000000001},"min":{"1min":0.16500000000000001,"5min":0.161,"15min":0.159},"max":{"1min":0.41299999999999998,"5min":0.439,"15min":0.45800000000000002},"last":0.217}]},{"osd":2,"last update":"Sun Dec 24 21:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.217,"5min":0.247,"15min":0.24099999999999999},"min":{"1min":0.157,"5min":0.157,"15min":0.157},"max":{"1min":0.28399999999999997,"5min":0.45600000000000002,"15min":0.46999999999999997},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.252,"5min":0.26900000000000002,"15min":0.26400000000000001},"min":{"1min":0.192,"5min":0.192,"15min":0.16900000000000001},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.23899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:06:26.355 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:26.356 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:26.356 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:26.356 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:26.356 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:26.356 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:26.356 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:26.357 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:26.357 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:26.357 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:26.357 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:26.357 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:26.357 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:26.357 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.7 2023-12-24T21:06:26.459 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:26.466+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:26.622 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:26.629+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:26.699 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.7 on osd.1 to deep-scrub 2023-12-24T21:06:26.711 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.6 2023-12-24T21:06:27.057 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.6 on osd.1 to deep-scrub 2023-12-24T21:06:27.069 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2023-12-24T21:06:27.246 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:27.253+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:27.399 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2023-12-24T21:06:27.409 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-12-24T21:06:27.445 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:27.452+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:27.635 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:27.641+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:27.780 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-12-24T21:06:27.792 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.0 2023-12-24T21:06:28.151 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.0 on osd.2 to deep-scrub 2023-12-24T21:06:28.163 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-12-24T21:06:28.213 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:28.220+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:28.438 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:28.445+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:28.491 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-12-24T21:06:28.503 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.0 2023-12-24T21:06:28.643 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:28.650+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:28.869 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 1.0 on osd.1 to deep-scrub 2023-12-24T21:06:28.879 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.3 2023-12-24T21:06:29.203 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.3 on osd.1 to deep-scrub 2023-12-24T21:06:29.215 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-12-24T21:06:29.230 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:29.237+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:29.433 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:29.440+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:29.593 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-12-24T21:06:29.604 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.1 2023-12-24T21:06:29.643 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:29.650+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:29.982 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.1 on osd.2 to deep-scrub 2023-12-24T21:06:29.993 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-12-24T21:06:30.197 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:30.204+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:30.367 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-12-24T21:06:30.379 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.2 2023-12-24T21:06:30.478 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:30.485+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:30.608 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:30.615+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:30.749 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.2 on osd.0 to deep-scrub 2023-12-24T21:06:30.760 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.4 2023-12-24T21:06:31.117 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.4 on osd.1 to deep-scrub 2023-12-24T21:06:31.129 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:06:31.155 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:31.162+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:31.473 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:31.480+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:31.585 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:31.592+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:32.166 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:32.173+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:32.513 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:32.520+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:32.552 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:32.560+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:33.196 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:33.203+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:33.548 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:33.555+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:33.587 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:33.594+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:34.246 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:34.253+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:34.505 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:34.512+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:34.543 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:34.550+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:35.284 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:35.291+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:35.520 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:35.527+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:35.521 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:35.528+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:36.316 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:36.323+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:36.535 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:36.542+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:36.561 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:36.568+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:37.307 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:37.314+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:37.513 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:37.520+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:37.558 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:37.565+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:38.329 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:38.336+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:38.545 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:38.552+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:38.572 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:38.579+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:39.347 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:39.353+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:39.572 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:39.579+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:39.575 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:39.582+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:40.382 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:40.389+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:40.529 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:40.536+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:40.545 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:40.552+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:41.393 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:41.399+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:41.570 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:41.577+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:41.573 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:41.580+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:42.352 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:42.358+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:42.551 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:42.558+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:42.591 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:42.598+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:43.330 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:43.337+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:43.576 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:43.582+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:43.617 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:43.624+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:44.374 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:44.380+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:44.593 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:44.599+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:44.652 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:44.658+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:45.406 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:45.413+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:45.627 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:45.634+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:45.677 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:45.684+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:46.409 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:46.415+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:46.581 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:46.587+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:46.649 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:46.656+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:47.431 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:47.437+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:47.611 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:47.618+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:47.648 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:47.655+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:48.390 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:48.397+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:48.604 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:48.611+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:48.687 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:48.694+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:49.378 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:49.385+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:49.613 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:49.620+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:49.687 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:49.694+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:50.394 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:50.401+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:50.573 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:50.580+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:50.704 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:50.711+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:51.130 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:06:51.366 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:51.373+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:51.457 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:06:51.458 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:06:51.470 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":5834,"stamp":"2023-12-24T21:06:50.633522+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,2,6,12,516],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001744"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2825,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543752+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T19:00:52.543752+0000","last_peered":"2023-12-24T19:00:52.543752+0000","last_clean":"2023-12-24T19:00:52.543752+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T19:00:52.543752+0000","last_undegraded":"2023-12-24T19:00:52.543752+0000","last_fullsized":"2023-12-24T19:00:52.543752+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3619,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543664+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T19:00:52.543664+0000","last_peered":"2023-12-24T19:00:52.543664+0000","last_clean":"2023-12-24T19:00:52.543664+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T19:00:52.543664+0000","last_undegraded":"2023-12-24T19:00:52.543664+0000","last_fullsized":"2023-12-24T19:00:52.543664+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2982,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543574+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T19:00:52.543574+0000","last_peered":"2023-12-24T19:00:52.543574+0000","last_clean":"2023-12-24T19:00:52.543574+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T19:00:52.543574+0000","last_undegraded":"2023-12-24T19:00:52.543574+0000","last_fullsized":"2023-12-24T19:00:52.543574+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10082,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:34.512915+0000","last_change":"2023-12-24T21:06:34.512915+0000","last_active":"2023-12-24T21:06:34.512915+0000","last_peered":"2023-12-24T21:06:34.512915+0000","last_clean":"2023-12-24T21:06:34.512915+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:06:34.512915+0000","last_undegraded":"2023-12-24T21:06:34.512915+0000","last_fullsized":"2023-12-24T21:06:34.512915+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3924,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:04:44.168071+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:04:44.168071+0000","last_peered":"2023-12-24T21:04:44.168071+0000","last_clean":"2023-12-24T21:04:44.168071+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:04:44.168071+0000","last_undegraded":"2023-12-24T21:04:44.168071+0000","last_fullsized":"2023-12-24T21:04:44.168071+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11204,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543302+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T19:00:52.543302+0000","last_peered":"2023-12-24T19:00:52.543302+0000","last_clean":"2023-12-24T19:00:52.543302+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T19:00:52.543302+0000","last_undegraded":"2023-12-24T19:00:52.543302+0000","last_fullsized":"2023-12-24T19:00:52.543302+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":285,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543005+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T19:00:52.543005+0000","last_peered":"2023-12-24T19:00:52.543005+0000","last_clean":"2023-12-24T19:00:52.543005+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T19:00:52.543005+0000","last_undegraded":"2023-12-24T19:00:52.543005+0000","last_fullsized":"2023-12-24T19:00:52.543005+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3530,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543393+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T19:00:52.543393+0000","last_peered":"2023-12-24T19:00:52.543393+0000","last_clean":"2023-12-24T19:00:52.543393+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T19:00:52.543393+0000","last_undegraded":"2023-12-24T19:00:52.543393+0000","last_fullsized":"2023-12-24T19:00:52.543393+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10732,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543086+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T19:00:52.543086+0000","last_peered":"2023-12-24T19:00:52.543086+0000","last_clean":"2023-12-24T19:00:52.543086+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T19:00:52.543086+0000","last_undegraded":"2023-12-24T19:00:52.543086+0000","last_fullsized":"2023-12-24T19:00:52.543086+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3508,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:04:46.246536+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:04:46.246536+0000","last_peered":"2023-12-24T21:04:46.246536+0000","last_clean":"2023-12-24T21:04:46.246536+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:04:46.246536+0000","last_undegraded":"2023-12-24T21:04:46.246536+0000","last_fullsized":"2023-12-24T21:04:46.246536+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 103s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12790,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:46.657679+0000","last_change":"2023-12-24T21:06:46.657679+0000","last_active":"2023-12-24T21:06:46.657679+0000","last_peered":"2023-12-24T21:06:46.657679+0000","last_clean":"2023-12-24T21:06:46.657679+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:06:46.657679+0000","last_undegraded":"2023-12-24T21:06:46.657679+0000","last_fullsized":"2023-12-24T21:06:46.657679+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3063,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:32.521716+0000","last_change":"2023-12-24T21:06:32.521716+0000","last_active":"2023-12-24T21:06:32.521716+0000","last_peered":"2023-12-24T21:06:32.521716+0000","last_clean":"2023-12-24T21:06:32.521716+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:06:32.521716+0000","last_undegraded":"2023-12-24T21:06:32.521716+0000","last_fullsized":"2023-12-24T21:06:32.521716+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3314,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543485+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T19:00:52.543485+0000","last_peered":"2023-12-24T19:00:52.543485+0000","last_clean":"2023-12-24T19:00:52.543485+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T19:00:52.543485+0000","last_undegraded":"2023-12-24T19:00:52.543485+0000","last_fullsized":"2023-12-24T19:00:52.543485+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740644,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,2,6,12,4],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:05:56 2023","interfaces":[{"interface":"back","average":{"1min":0.216,"5min":0.21299999999999999,"15min":0.215},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14099999999999999},"max":{"1min":0.38900000000000001,"5min":0.40000000000000002,"15min":0.47499999999999998},"last":0.17799999999999999},{"interface":"front","average":{"1min":0.23799999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.16500000000000001},"max":{"1min":0.42999999999999999,"5min":0.45600000000000002,"15min":0.48699999999999999},"last":0.19900000000000001}]},{"osd":1,"last update":"Sun Dec 24 21:05:56 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.24099999999999999,"15min":0.23599999999999999},"min":{"1min":0.19,"5min":0.182,"15min":0.16400000000000001},"max":{"1min":0.44900000000000001,"5min":0.46300000000000002,"15min":0.46300000000000002},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.26800000000000002,"5min":0.26300000000000001,"15min":0.26200000000000001},"min":{"1min":0.20799999999999999,"5min":0.182,"15min":0.17799999999999999},"max":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.49299999999999999},"last":0.23999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359740643,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":0.19600000000000001,"15min":0.20100000000000001},"min":{"1min":0.155,"5min":0.14299999999999999,"15min":0.127},"max":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.44800000000000001},"last":0.16900000000000001},{"interface":"front","average":{"1min":0.23300000000000001,"5min":0.22700000000000001,"15min":0.22900000000000001},"min":{"1min":0.183,"5min":0.155,"15min":0.14499999999999999},"max":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47599999999999998},"last":0.223}]},{"osd":2,"last update":"Sun Dec 24 21:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":0.249,"15min":0.249},"min":{"1min":0.186,"5min":0.17100000000000001,"15min":0.16800000000000001},"max":{"1min":0.438,"5min":0.48799999999999999,"15min":0.48799999999999999},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.28699999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"min":{"1min":0.23400000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.495,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.28599999999999998}]}]},{"osd":1,"up_from":8,"seq":34359740646,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.18099999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"min":{"1min":0.14599999999999999,"5min":0.13300000000000001,"15min":0.13300000000000001},"max":{"1min":0.437,"5min":0.437,"15min":0.437},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.20799999999999999,"5min":0.22700000000000001,"15min":0.22800000000000001},"min":{"1min":0.16500000000000001,"5min":0.161,"15min":0.159},"max":{"1min":0.41299999999999998,"5min":0.439,"15min":0.45800000000000002},"last":0.35899999999999999}]},{"osd":2,"last update":"Sun Dec 24 21:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.217,"5min":0.247,"15min":0.24099999999999999},"min":{"1min":0.157,"5min":0.157,"15min":0.157},"max":{"1min":0.28399999999999997,"5min":0.45600000000000002,"15min":0.46999999999999997},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.252,"5min":0.26900000000000002,"15min":0.26400000000000001},"min":{"1min":0.192,"5min":0.192,"15min":0.16900000000000001},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.44}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:06:51.471 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:51.471 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:51.472 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:51.472 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:51.472 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:51.472 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:51.472 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:51.472 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:51.473 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:51.473 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:51.473 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:51.473 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:51.473 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:06:51.473 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:06:51.526 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:51.533+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:51.719 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:51.726+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:52.399 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:52.406+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:52.478 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:52.486+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:52.689 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:52.696+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:53.384 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:53.391+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:53.476 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:53.484+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:53.707 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:53.715+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:54.386 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:54.394+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:54.520 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:54.528+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:54.718 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:54.726+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:55.366 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:55.373+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:55.489 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:55.496+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:55.734 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:55.742+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:56.366 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:56.374+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:56.470 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:56.478+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:56.764 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:56.772+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:57.395 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:57.403+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:57.436 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:57.444+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:57.808 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:57.816+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:58.376 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:58.384+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:58.474 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:58.482+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:58.803 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:58.811+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:59.403 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:06:59.410+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:59.506 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:06:59.514+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:06:59.762 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:06:59.769+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:00.398 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:00.405+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:00.494 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:00.501+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:00.732 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:00.740+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:01.370 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:01.378+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:01.492 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:01.500+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:01.696 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:01.704+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:02.368 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:02.375+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:02.467 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:02.475+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:02.680 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:02.687+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:03.341 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:03.348+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:03.464 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:03.472+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:03.675 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:03.682+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:04.314 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:04.322+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:04.448 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:04.455+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:04.681 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:04.688+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:05.354 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:05.361+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:05.418 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:05.425+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:05.666 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:05.672+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:06.318 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:06.324+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:06.454 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:06.461+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:06.710 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:06.717+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:07.331 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:07.337+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:07.490 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:07.496+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:07.725 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:07.731+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:08.374 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:08.380+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:08.452 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:08.459+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:08.712 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:08.718+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:09.329 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:09.335+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:09.429 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:09.435+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:09.688 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:09.695+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:10.283 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:10.289+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:10.461 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:10.467+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:10.666 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:10.672+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:11.271 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:11.277+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:11.463 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:11.469+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:11.474 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:07:11.651 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:11.657+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:11.854 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:07:11.856 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:07:11.868 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":5844,"stamp":"2023-12-24T21:07:10.636315+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,2,6,12,516],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001804"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2825,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543752+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T19:00:52.543752+0000","last_peered":"2023-12-24T19:00:52.543752+0000","last_clean":"2023-12-24T19:00:52.543752+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T19:00:52.543752+0000","last_undegraded":"2023-12-24T19:00:52.543752+0000","last_fullsized":"2023-12-24T19:00:52.543752+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3619,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543664+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T19:00:52.543664+0000","last_peered":"2023-12-24T19:00:52.543664+0000","last_clean":"2023-12-24T19:00:52.543664+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T19:00:52.543664+0000","last_undegraded":"2023-12-24T19:00:52.543664+0000","last_fullsized":"2023-12-24T19:00:52.543664+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2982,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543574+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T19:00:52.543574+0000","last_peered":"2023-12-24T19:00:52.543574+0000","last_clean":"2023-12-24T19:00:52.543574+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T19:00:52.543574+0000","last_undegraded":"2023-12-24T19:00:52.543574+0000","last_fullsized":"2023-12-24T19:00:52.543574+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10088,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:58.810853+0000","last_change":"2023-12-24T21:06:58.810853+0000","last_active":"2023-12-24T21:06:58.810853+0000","last_peered":"2023-12-24T21:06:58.810853+0000","last_clean":"2023-12-24T21:06:58.810853+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:06:58.810853+0000","last_undegraded":"2023-12-24T21:06:58.810853+0000","last_fullsized":"2023-12-24T21:06:58.810853+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3924,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:04:44.168071+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:04:44.168071+0000","last_peered":"2023-12-24T21:04:44.168071+0000","last_clean":"2023-12-24T21:04:44.168071+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:04:44.168071+0000","last_undegraded":"2023-12-24T21:04:44.168071+0000","last_fullsized":"2023-12-24T21:04:44.168071+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11204,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543302+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T19:00:52.543302+0000","last_peered":"2023-12-24T19:00:52.543302+0000","last_clean":"2023-12-24T19:00:52.543302+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T19:00:52.543302+0000","last_undegraded":"2023-12-24T19:00:52.543302+0000","last_fullsized":"2023-12-24T19:00:52.543302+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":285,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543005+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T19:00:52.543005+0000","last_peered":"2023-12-24T19:00:52.543005+0000","last_clean":"2023-12-24T19:00:52.543005+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T19:00:52.543005+0000","last_undegraded":"2023-12-24T19:00:52.543005+0000","last_fullsized":"2023-12-24T19:00:52.543005+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3530,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543393+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T19:00:52.543393+0000","last_peered":"2023-12-24T19:00:52.543393+0000","last_clean":"2023-12-24T19:00:52.543393+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T19:00:52.543393+0000","last_undegraded":"2023-12-24T19:00:52.543393+0000","last_fullsized":"2023-12-24T19:00:52.543393+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10732,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543086+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T19:00:52.543086+0000","last_peered":"2023-12-24T19:00:52.543086+0000","last_clean":"2023-12-24T19:00:52.543086+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T19:00:52.543086+0000","last_undegraded":"2023-12-24T19:00:52.543086+0000","last_fullsized":"2023-12-24T19:00:52.543086+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3508,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:04:46.246536+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:04:46.246536+0000","last_peered":"2023-12-24T21:04:46.246536+0000","last_clean":"2023-12-24T21:04:46.246536+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:04:46.246536+0000","last_undegraded":"2023-12-24T21:04:46.246536+0000","last_fullsized":"2023-12-24T21:04:46.246536+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 103s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12796,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:07:00.740102+0000","last_change":"2023-12-24T21:07:00.740102+0000","last_active":"2023-12-24T21:07:00.740102+0000","last_peered":"2023-12-24T21:07:00.740102+0000","last_clean":"2023-12-24T21:07:00.740102+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:07:00.740102+0000","last_undegraded":"2023-12-24T21:07:00.740102+0000","last_fullsized":"2023-12-24T21:07:00.740102+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3072,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:07:02.687072+0000","last_change":"2023-12-24T21:07:02.687072+0000","last_active":"2023-12-24T21:07:02.687072+0000","last_peered":"2023-12-24T21:07:02.687072+0000","last_clean":"2023-12-24T21:07:02.687072+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:07:02.687072+0000","last_undegraded":"2023-12-24T21:07:02.687072+0000","last_fullsized":"2023-12-24T21:07:02.687072+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3314,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543485+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T19:00:52.543485+0000","last_peered":"2023-12-24T19:00:52.543485+0000","last_clean":"2023-12-24T19:00:52.543485+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T19:00:52.543485+0000","last_undegraded":"2023-12-24T19:00:52.543485+0000","last_fullsized":"2023-12-24T19:00:52.543485+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740648,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,2,6,12,4],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.217,"5min":0.21099999999999999,"15min":0.21299999999999999},"min":{"1min":0.14899999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.47499999999999998},"last":0.19},{"interface":"front","average":{"1min":0.248,"5min":0.24099999999999999,"15min":0.23999999999999999},"min":{"1min":0.17899999999999999,"5min":0.17499999999999999,"15min":0.16500000000000001},"max":{"1min":0.38,"5min":0.45600000000000002,"15min":0.48699999999999999},"last":0.24299999999999999}]},{"osd":1,"last update":"Sun Dec 24 21:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.246,"5min":0.23999999999999999,"15min":0.23499999999999999},"min":{"1min":0.19600000000000001,"5min":0.182,"15min":0.16600000000000001},"max":{"1min":0.40799999999999997,"5min":0.44900000000000001,"15min":0.46300000000000002},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.26200000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"min":{"1min":0.191,"5min":0.191,"15min":0.17799999999999999},"max":{"1min":0.41899999999999998,"5min":0.47199999999999998,"15min":0.47599999999999998},"last":0.20999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359740647,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:07:05 2023","interfaces":[{"interface":"back","average":{"1min":0.184,"5min":0.192,"15min":0.20000000000000001},"min":{"1min":0.156,"5min":0.14299999999999999,"15min":0.127},"max":{"1min":0.23400000000000001,"5min":0.41099999999999998,"15min":0.44800000000000001},"last":0.161},{"interface":"front","average":{"1min":0.224,"5min":0.22500000000000001,"15min":0.22800000000000001},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.14499999999999999},"max":{"1min":0.28399999999999997,"5min":0.47199999999999998,"15min":0.47599999999999998},"last":0.21099999999999999}]},{"osd":2,"last update":"Sun Dec 24 21:07:05 2023","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":0.249,"15min":0.25},"min":{"1min":0.20399999999999999,"5min":0.17100000000000001,"15min":0.16800000000000001},"max":{"1min":0.34799999999999998,"5min":0.438,"15min":0.48799999999999999},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.27400000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":0.32700000000000001,"5min":0.495,"15min":0.51300000000000001},"last":0.248}]}]},{"osd":1,"up_from":8,"seq":34359740650,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:07:05 2023","interfaces":[{"interface":"back","average":{"1min":0.19700000000000001,"5min":0.19500000000000001,"15min":0.20000000000000001},"min":{"1min":0.126,"5min":0.126,"15min":0.126},"max":{"1min":0.32800000000000001,"5min":0.437,"15min":0.437},"last":0.17499999999999999},{"interface":"front","average":{"1min":0.22900000000000001,"5min":0.22600000000000001,"15min":0.22700000000000001},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":0.35899999999999999,"5min":0.439,"15min":0.45800000000000002},"last":0.20300000000000001}]},{"osd":2,"last update":"Sun Dec 24 21:07:05 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.24399999999999999,"15min":0.24099999999999999},"min":{"1min":0.16400000000000001,"5min":0.157,"15min":0.157},"max":{"1min":0.42699999999999999,"5min":0.45600000000000002,"15min":0.46999999999999997},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.26800000000000002,"5min":0.26700000000000002,"15min":0.26400000000000001},"min":{"1min":0.20699999999999999,"5min":0.192,"15min":0.16900000000000001},"max":{"1min":0.44,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.23999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:07:11.870 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:11.870 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:11.870 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:11.870 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:11.871 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:11.871 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:11.871 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:11.871 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:11.871 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:11.872 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:11.872 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:11.872 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:11.872 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:11.872 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:07:12.289 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:12.295+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:12.440 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:12.447+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:12.639 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:12.646+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:13.336 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:13.342+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:13.441 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:13.448+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:13.651 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:13.657+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:14.366 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:14.372+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:14.468 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:14.474+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:14.699 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:14.706+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:15.412 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:15.418+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:15.478 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:15.484+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:15.746 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:15.752+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:16.420 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:16.426+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:16.522 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:16.528+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:16.796 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:16.803+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:17.403 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:17.408+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:17.513 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:17.519+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:17.772 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:17.778+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:18.420 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:18.426+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:18.527 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:18.534+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:18.725 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:18.731+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:19.381 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:19.387+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:19.485 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:19.491+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:19.738 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:19.744+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:20.376 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:20.382+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:20.535 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:20.541+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:20.702 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:20.709+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:21.376 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:21.381+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:21.540 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:21.546+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:21.659 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:21.665+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:22.345 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:22.351+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:22.519 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:22.525+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:22.671 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:22.677+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:23.313 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:23.318+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:23.498 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:23.504+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:23.648 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:23.655+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:24.332 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:24.338+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:24.468 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:24.474+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:24.691 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:24.697+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:25.316 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:25.322+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:25.433 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:25.439+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:25.740 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:25.746+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:26.303 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:26.309+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:26.452 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:26.458+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:26.708 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:26.714+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:27.260 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:27.266+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:27.437 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:27.443+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:27.661 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:27.667+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:28.226 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:28.232+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:28.404 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:28.410+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:28.615 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:28.621+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:29.271 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:29.277+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:29.410 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:29.416+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:29.586 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:29.592+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:30.269 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:30.275+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:30.439 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:30.444+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:30.563 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:30.569+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:31.302 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:31.308+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:31.430 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:31.436+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:31.582 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:31.588+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:31.873 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:07:32.214 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:07:32.217 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:07:32.227 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":5854,"stamp":"2023-12-24T21:07:30.639100+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,2,6,12,516],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001704"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2825,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543752+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T19:00:52.543752+0000","last_peered":"2023-12-24T19:00:52.543752+0000","last_clean":"2023-12-24T19:00:52.543752+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T19:00:52.543752+0000","last_undegraded":"2023-12-24T19:00:52.543752+0000","last_fullsized":"2023-12-24T19:00:52.543752+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3619,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543664+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T19:00:52.543664+0000","last_peered":"2023-12-24T19:00:52.543664+0000","last_clean":"2023-12-24T19:00:52.543664+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T19:00:52.543664+0000","last_undegraded":"2023-12-24T19:00:52.543664+0000","last_fullsized":"2023-12-24T19:00:52.543664+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2982,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543574+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T19:00:52.543574+0000","last_peered":"2023-12-24T19:00:52.543574+0000","last_clean":"2023-12-24T19:00:52.543574+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T19:00:52.543574+0000","last_undegraded":"2023-12-24T19:00:52.543574+0000","last_fullsized":"2023-12-24T19:00:52.543574+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10091,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:07:09.696058+0000","last_change":"2023-12-24T21:07:09.696058+0000","last_active":"2023-12-24T21:07:09.696058+0000","last_peered":"2023-12-24T21:07:09.696058+0000","last_clean":"2023-12-24T21:07:09.696058+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:07:09.696058+0000","last_undegraded":"2023-12-24T21:07:09.696058+0000","last_fullsized":"2023-12-24T21:07:09.696058+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3924,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:04:44.168071+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:04:44.168071+0000","last_peered":"2023-12-24T21:04:44.168071+0000","last_clean":"2023-12-24T21:04:44.168071+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:04:44.168071+0000","last_undegraded":"2023-12-24T21:04:44.168071+0000","last_fullsized":"2023-12-24T21:04:44.168071+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11204,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543302+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T19:00:52.543302+0000","last_peered":"2023-12-24T19:00:52.543302+0000","last_clean":"2023-12-24T19:00:52.543302+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T19:00:52.543302+0000","last_undegraded":"2023-12-24T19:00:52.543302+0000","last_fullsized":"2023-12-24T19:00:52.543302+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":285,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543005+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T19:00:52.543005+0000","last_peered":"2023-12-24T19:00:52.543005+0000","last_clean":"2023-12-24T19:00:52.543005+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T19:00:52.543005+0000","last_undegraded":"2023-12-24T19:00:52.543005+0000","last_fullsized":"2023-12-24T19:00:52.543005+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3530,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543393+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T19:00:52.543393+0000","last_peered":"2023-12-24T19:00:52.543393+0000","last_clean":"2023-12-24T19:00:52.543393+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T19:00:52.543393+0000","last_undegraded":"2023-12-24T19:00:52.543393+0000","last_fullsized":"2023-12-24T19:00:52.543393+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10732,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543086+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T19:00:52.543086+0000","last_peered":"2023-12-24T19:00:52.543086+0000","last_clean":"2023-12-24T19:00:52.543086+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T19:00:52.543086+0000","last_undegraded":"2023-12-24T19:00:52.543086+0000","last_fullsized":"2023-12-24T19:00:52.543086+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3508,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:04:46.246536+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:04:46.246536+0000","last_peered":"2023-12-24T21:04:46.246536+0000","last_clean":"2023-12-24T21:04:46.246536+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:04:46.246536+0000","last_undegraded":"2023-12-24T21:04:46.246536+0000","last_fullsized":"2023-12-24T21:04:46.246536+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 103s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12799,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:07:11.657428+0000","last_change":"2023-12-24T21:07:11.657428+0000","last_active":"2023-12-24T21:07:11.657428+0000","last_peered":"2023-12-24T21:07:11.657428+0000","last_clean":"2023-12-24T21:07:11.657428+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:07:11.657428+0000","last_undegraded":"2023-12-24T21:07:11.657428+0000","last_fullsized":"2023-12-24T21:07:11.657428+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3075,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:07:14.707106+0000","last_change":"2023-12-24T21:07:14.707106+0000","last_active":"2023-12-24T21:07:14.707106+0000","last_peered":"2023-12-24T21:07:14.707106+0000","last_clean":"2023-12-24T21:07:14.707106+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:07:14.707106+0000","last_undegraded":"2023-12-24T21:07:14.707106+0000","last_fullsized":"2023-12-24T21:07:14.707106+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3314,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543485+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T19:00:52.543485+0000","last_peered":"2023-12-24T19:00:52.543485+0000","last_clean":"2023-12-24T19:00:52.543485+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T19:00:52.543485+0000","last_undegraded":"2023-12-24T19:00:52.543485+0000","last_fullsized":"2023-12-24T19:00:52.543485+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740652,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,2,6,12,4],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.217,"5min":0.21099999999999999,"15min":0.21299999999999999},"min":{"1min":0.14899999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.47499999999999998},"last":0.20200000000000001},{"interface":"front","average":{"1min":0.248,"5min":0.24099999999999999,"15min":0.23999999999999999},"min":{"1min":0.17899999999999999,"5min":0.17499999999999999,"15min":0.16500000000000001},"max":{"1min":0.38,"5min":0.45600000000000002,"15min":0.48699999999999999},"last":0.184}]},{"osd":1,"last update":"Sun Dec 24 21:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.246,"5min":0.23999999999999999,"15min":0.23499999999999999},"min":{"1min":0.19600000000000001,"5min":0.182,"15min":0.16600000000000001},"max":{"1min":0.40799999999999997,"5min":0.44900000000000001,"15min":0.46300000000000002},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.26200000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"min":{"1min":0.191,"5min":0.191,"15min":0.17799999999999999},"max":{"1min":0.41899999999999998,"5min":0.47199999999999998,"15min":0.47599999999999998},"last":0.219}]}]},{"osd":0,"up_from":8,"seq":34359740651,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:07:05 2023","interfaces":[{"interface":"back","average":{"1min":0.184,"5min":0.192,"15min":0.20000000000000001},"min":{"1min":0.156,"5min":0.14299999999999999,"15min":0.127},"max":{"1min":0.23400000000000001,"5min":0.41099999999999998,"15min":0.44800000000000001},"last":0.17499999999999999},{"interface":"front","average":{"1min":0.224,"5min":0.22500000000000001,"15min":0.22800000000000001},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.14499999999999999},"max":{"1min":0.28399999999999997,"5min":0.47199999999999998,"15min":0.47599999999999998},"last":0.214}]},{"osd":2,"last update":"Sun Dec 24 21:07:05 2023","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":0.249,"15min":0.25},"min":{"1min":0.20399999999999999,"5min":0.17100000000000001,"15min":0.16800000000000001},"max":{"1min":0.34799999999999998,"5min":0.438,"15min":0.48799999999999999},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.27400000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":0.32700000000000001,"5min":0.495,"15min":0.51300000000000001},"last":0.34899999999999998}]}]},{"osd":1,"up_from":8,"seq":34359740654,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:07:05 2023","interfaces":[{"interface":"back","average":{"1min":0.19700000000000001,"5min":0.19500000000000001,"15min":0.20000000000000001},"min":{"1min":0.126,"5min":0.126,"15min":0.126},"max":{"1min":0.32800000000000001,"5min":0.437,"15min":0.437},"last":0.17699999999999999},{"interface":"front","average":{"1min":0.22900000000000001,"5min":0.22600000000000001,"15min":0.22700000000000001},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":0.35899999999999999,"5min":0.439,"15min":0.45800000000000002},"last":0.21099999999999999}]},{"osd":2,"last update":"Sun Dec 24 21:07:05 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.24399999999999999,"15min":0.24099999999999999},"min":{"1min":0.16400000000000001,"5min":0.157,"15min":0.157},"max":{"1min":0.42699999999999999,"5min":0.45600000000000002,"15min":0.46999999999999997},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.26800000000000002,"5min":0.26700000000000002,"15min":0.26400000000000001},"min":{"1min":0.20699999999999999,"5min":0.192,"15min":0.16900000000000001},"max":{"1min":0.44,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.24099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:07:32.228 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:32.229 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:32.229 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:32.229 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:32.229 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:32.229 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:32.229 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:32.230 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:32.230 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:32.230 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:32.230 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:32.230 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:32.230 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:32.231 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:07:32.280 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:32.286+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:32.423 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:32.429+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:32.631 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:32.637+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:33.271 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:33.277+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:33.463 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:33.469+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:33.640 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:33.646+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:34.266 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:34.272+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:34.419 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:34.425+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:34.617 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:34.623+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:35.221 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:35.227+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:35.371 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:35.377+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:35.657 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:35.663+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:36.257 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:36.264+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:36.367 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:36.372+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:36.696 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:36.702+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:37.222 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:37.229+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:37.361 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:37.368+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:37.697 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:37.704+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:38.188 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:38.195+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:38.327 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:38.334+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:38.671 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:38.677+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:39.172 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:39.179+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:39.282 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:39.289+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:39.659 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:39.666+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:40.178 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:40.184+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:40.265 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:40.272+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:40.616 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:40.623+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:41.157 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:41.164+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:41.269 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:41.276+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:41.572 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:41.579+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:42.189 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:42.196+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:42.274 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:42.281+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:42.575 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:42.582+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:43.234 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:43.241+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:43.295 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:43.302+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:43.559 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:43.566+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:44.237 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:44.243+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:44.272 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:44.279+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:44.517 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:44.524+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:45.258 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:45.264+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:45.321 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:45.328+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:45.503 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:45.510+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:46.212 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:46.219+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:46.345 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:46.352+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:46.462 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:46.469+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:47.188 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:47.194+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:47.332 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:47.339+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:47.422 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:47.429+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:48.232 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:48.239+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:48.347 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:48.354+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:48.387 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:48.394+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:49.256 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:49.263+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:49.375 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:49.382+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:49.416 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:49.423+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:50.221 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:50.228+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:50.379 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:50.385+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:50.397 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:50.403+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:51.190 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:51.196+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:51.369 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:51.374+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:51.381 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:51.386+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:52.222 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:52.228+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:52.232 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:07:52.334 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:52.340+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:52.343 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:52.349+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:52.559 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:07:52.560 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:07:52.572 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":5864,"stamp":"2023-12-24T21:07:50.641930+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,2,6,10,518],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001749"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2825,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543752+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T19:00:52.543752+0000","last_peered":"2023-12-24T19:00:52.543752+0000","last_clean":"2023-12-24T19:00:52.543752+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T19:00:52.543752+0000","last_undegraded":"2023-12-24T19:00:52.543752+0000","last_fullsized":"2023-12-24T19:00:52.543752+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3619,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543664+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T19:00:52.543664+0000","last_peered":"2023-12-24T19:00:52.543664+0000","last_clean":"2023-12-24T19:00:52.543664+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T19:00:52.543664+0000","last_undegraded":"2023-12-24T19:00:52.543664+0000","last_fullsized":"2023-12-24T19:00:52.543664+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2982,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543574+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T19:00:52.543574+0000","last_peered":"2023-12-24T19:00:52.543574+0000","last_clean":"2023-12-24T19:00:52.543574+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T19:00:52.543574+0000","last_undegraded":"2023-12-24T19:00:52.543574+0000","last_fullsized":"2023-12-24T19:00:52.543574+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10097,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:07:38.677500+0000","last_change":"2023-12-24T21:07:38.677500+0000","last_active":"2023-12-24T21:07:38.677500+0000","last_peered":"2023-12-24T21:07:38.677500+0000","last_clean":"2023-12-24T21:07:38.677500+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:07:38.677500+0000","last_undegraded":"2023-12-24T21:07:38.677500+0000","last_fullsized":"2023-12-24T21:07:38.677500+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3924,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:04:44.168071+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:04:44.168071+0000","last_peered":"2023-12-24T21:04:44.168071+0000","last_clean":"2023-12-24T21:04:44.168071+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:04:44.168071+0000","last_undegraded":"2023-12-24T21:04:44.168071+0000","last_fullsized":"2023-12-24T21:04:44.168071+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11204,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543302+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T19:00:52.543302+0000","last_peered":"2023-12-24T19:00:52.543302+0000","last_clean":"2023-12-24T19:00:52.543302+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T19:00:52.543302+0000","last_undegraded":"2023-12-24T19:00:52.543302+0000","last_fullsized":"2023-12-24T19:00:52.543302+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":285,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543005+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T19:00:52.543005+0000","last_peered":"2023-12-24T19:00:52.543005+0000","last_clean":"2023-12-24T19:00:52.543005+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T19:00:52.543005+0000","last_undegraded":"2023-12-24T19:00:52.543005+0000","last_fullsized":"2023-12-24T19:00:52.543005+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3530,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543393+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T19:00:52.543393+0000","last_peered":"2023-12-24T19:00:52.543393+0000","last_clean":"2023-12-24T19:00:52.543393+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T19:00:52.543393+0000","last_undegraded":"2023-12-24T19:00:52.543393+0000","last_fullsized":"2023-12-24T19:00:52.543393+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10732,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543086+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T19:00:52.543086+0000","last_peered":"2023-12-24T19:00:52.543086+0000","last_clean":"2023-12-24T19:00:52.543086+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T19:00:52.543086+0000","last_undegraded":"2023-12-24T19:00:52.543086+0000","last_fullsized":"2023-12-24T19:00:52.543086+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3508,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:04:46.246536+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:04:46.246536+0000","last_peered":"2023-12-24T21:04:46.246536+0000","last_clean":"2023-12-24T21:04:46.246536+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:04:46.246536+0000","last_undegraded":"2023-12-24T21:04:46.246536+0000","last_fullsized":"2023-12-24T21:04:46.246536+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 103s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12805,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:07:44.524412+0000","last_change":"2023-12-24T21:07:44.524412+0000","last_active":"2023-12-24T21:07:44.524412+0000","last_peered":"2023-12-24T21:07:44.524412+0000","last_clean":"2023-12-24T21:07:44.524412+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:07:44.524412+0000","last_undegraded":"2023-12-24T21:07:44.524412+0000","last_fullsized":"2023-12-24T21:07:44.524412+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3078,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:07:33.647833+0000","last_change":"2023-12-24T21:07:33.647833+0000","last_active":"2023-12-24T21:07:33.647833+0000","last_peered":"2023-12-24T21:07:33.647833+0000","last_clean":"2023-12-24T21:07:33.647833+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:07:33.647833+0000","last_undegraded":"2023-12-24T21:07:33.647833+0000","last_fullsized":"2023-12-24T21:07:33.647833+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3314,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543485+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T19:00:52.543485+0000","last_peered":"2023-12-24T19:00:52.543485+0000","last_clean":"2023-12-24T19:00:52.543485+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T19:00:52.543485+0000","last_undegraded":"2023-12-24T19:00:52.543485+0000","last_fullsized":"2023-12-24T19:00:52.543485+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740656,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,2,6,10,6],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.217,"5min":0.21099999999999999,"15min":0.21299999999999999},"min":{"1min":0.14899999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.47499999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":0.248,"5min":0.24099999999999999,"15min":0.23999999999999999},"min":{"1min":0.17899999999999999,"5min":0.17499999999999999,"15min":0.16500000000000001},"max":{"1min":0.38,"5min":0.45600000000000002,"15min":0.48699999999999999},"last":0.187}]},{"osd":1,"last update":"Sun Dec 24 21:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.246,"5min":0.23999999999999999,"15min":0.23499999999999999},"min":{"1min":0.19600000000000001,"5min":0.182,"15min":0.16600000000000001},"max":{"1min":0.40799999999999997,"5min":0.44900000000000001,"15min":0.46300000000000002},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.26200000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"min":{"1min":0.191,"5min":0.191,"15min":0.17799999999999999},"max":{"1min":0.41899999999999998,"5min":0.47199999999999998,"15min":0.47599999999999998},"last":0.22}]}]},{"osd":0,"up_from":8,"seq":34359740655,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:07:05 2023","interfaces":[{"interface":"back","average":{"1min":0.184,"5min":0.192,"15min":0.20000000000000001},"min":{"1min":0.156,"5min":0.14299999999999999,"15min":0.127},"max":{"1min":0.23400000000000001,"5min":0.41099999999999998,"15min":0.44800000000000001},"last":0.16500000000000001},{"interface":"front","average":{"1min":0.224,"5min":0.22500000000000001,"15min":0.22800000000000001},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.14499999999999999},"max":{"1min":0.28399999999999997,"5min":0.47199999999999998,"15min":0.47599999999999998},"last":0.20799999999999999}]},{"osd":2,"last update":"Sun Dec 24 21:07:05 2023","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":0.249,"15min":0.25},"min":{"1min":0.20399999999999999,"5min":0.17100000000000001,"15min":0.16800000000000001},"max":{"1min":0.34799999999999998,"5min":0.438,"15min":0.48799999999999999},"last":0.17899999999999999},{"interface":"front","average":{"1min":0.27400000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":0.32700000000000001,"5min":0.495,"15min":0.51300000000000001},"last":0.23200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359740658,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:07:05 2023","interfaces":[{"interface":"back","average":{"1min":0.19700000000000001,"5min":0.19500000000000001,"15min":0.20000000000000001},"min":{"1min":0.126,"5min":0.126,"15min":0.126},"max":{"1min":0.32800000000000001,"5min":0.437,"15min":0.437},"last":0.17899999999999999},{"interface":"front","average":{"1min":0.22900000000000001,"5min":0.22600000000000001,"15min":0.22700000000000001},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":0.35899999999999999,"5min":0.439,"15min":0.45800000000000002},"last":0.21299999999999999}]},{"osd":2,"last update":"Sun Dec 24 21:07:05 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.24399999999999999,"15min":0.24099999999999999},"min":{"1min":0.16400000000000001,"5min":0.157,"15min":0.157},"max":{"1min":0.42699999999999999,"5min":0.45600000000000002,"15min":0.46999999999999997},"last":0.19800000000000001},{"interface":"front","average":{"1min":0.26800000000000002,"5min":0.26700000000000002,"15min":0.26400000000000001},"min":{"1min":0.20699999999999999,"5min":0.192,"15min":0.16900000000000001},"max":{"1min":0.44,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.249}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:07:52.575 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:52.575 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:52.575 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:52.575 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:52.576 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:52.576 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:52.576 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:52.576 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:52.576 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:52.576 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:52.577 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:52.577 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:52.577 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:07:52.577 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:07:53.191 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:53.196+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:53.302 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:53.308+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:53.345 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:53.351+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:54.207 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:54.209+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:54.297 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:54.303+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:54.337 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:54.343+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:55.207 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:55.213+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:55.324 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:55.329+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:55.353 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:55.359+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:56.245 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:56.250+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:56.331 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:56.337+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:56.342 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:56.348+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:57.226 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:57.232+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:57.359 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:57.364+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:57.367 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:57.373+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:58.241 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:58.246+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:58.329 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:58.335+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:58.396 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:58.401+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:59.249 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:07:59.255+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:59.327 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:07:59.333+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:07:59.437 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:07:59.442+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:00.200 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:00.206+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:00.355 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:00.361+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:00.425 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:00.431+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:01.199 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:01.204+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:01.308 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:01.313+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:01.399 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:01.405+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:02.233 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:02.239+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:02.316 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:02.321+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:02.433 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:02.438+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:03.252 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:03.257+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:03.326 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:03.331+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:03.431 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:03.437+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:04.293 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:04.298+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:04.294 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:04.299+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:04.456 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:04.462+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:05.276 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:05.281+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:05.330 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:05.335+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:05.503 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:05.508+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:06.273 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:06.279+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:06.301 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:06.307+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:06.459 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:06.464+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:07.294 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:07.299+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:07.334 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:07.339+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:07.462 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:07.467+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:08.304 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:08.309+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:08.307 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:08.312+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:08.481 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:08.486+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:09.256 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:09.261+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:09.258 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:09.263+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:09.503 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:09.509+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:10.239 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:10.245+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:10.244 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:10.250+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:10.545 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:10.549+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:11.214 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:11.219+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:11.256 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:11.261+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:11.561 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:11.565+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:12.213 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:12.216+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:12.267 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:12.273+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:12.519 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:12.524+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:12.580 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:08:12.956 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:08:12.959 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:08:12.971 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":5875,"stamp":"2023-12-24T21:08:12.645155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,4,6,10,518],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001816"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2825,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543752+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T19:00:52.543752+0000","last_peered":"2023-12-24T19:00:52.543752+0000","last_clean":"2023-12-24T19:00:52.543752+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T19:00:52.543752+0000","last_undegraded":"2023-12-24T19:00:52.543752+0000","last_fullsized":"2023-12-24T19:00:52.543752+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3619,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543664+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T19:00:52.543664+0000","last_peered":"2023-12-24T19:00:52.543664+0000","last_clean":"2023-12-24T19:00:52.543664+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T19:00:52.543664+0000","last_undegraded":"2023-12-24T19:00:52.543664+0000","last_fullsized":"2023-12-24T19:00:52.543664+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2982,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543574+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T19:00:52.543574+0000","last_peered":"2023-12-24T19:00:52.543574+0000","last_clean":"2023-12-24T19:00:52.543574+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T19:00:52.543574+0000","last_undegraded":"2023-12-24T19:00:52.543574+0000","last_fullsized":"2023-12-24T19:00:52.543574+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10103,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:01.314733+0000","last_change":"2023-12-24T21:08:01.314733+0000","last_active":"2023-12-24T21:08:01.314733+0000","last_peered":"2023-12-24T21:08:01.314733+0000","last_clean":"2023-12-24T21:08:01.314733+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:08:01.314733+0000","last_undegraded":"2023-12-24T21:08:01.314733+0000","last_fullsized":"2023-12-24T21:08:01.314733+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3924,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:04:44.168071+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:04:44.168071+0000","last_peered":"2023-12-24T21:04:44.168071+0000","last_clean":"2023-12-24T21:04:44.168071+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:04:44.168071+0000","last_undegraded":"2023-12-24T21:04:44.168071+0000","last_fullsized":"2023-12-24T21:04:44.168071+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11204,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543302+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T19:00:52.543302+0000","last_peered":"2023-12-24T19:00:52.543302+0000","last_clean":"2023-12-24T19:00:52.543302+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T19:00:52.543302+0000","last_undegraded":"2023-12-24T19:00:52.543302+0000","last_fullsized":"2023-12-24T19:00:52.543302+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":285,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543005+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T19:00:52.543005+0000","last_peered":"2023-12-24T19:00:52.543005+0000","last_clean":"2023-12-24T19:00:52.543005+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T19:00:52.543005+0000","last_undegraded":"2023-12-24T19:00:52.543005+0000","last_fullsized":"2023-12-24T19:00:52.543005+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3530,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543393+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T19:00:52.543393+0000","last_peered":"2023-12-24T19:00:52.543393+0000","last_clean":"2023-12-24T19:00:52.543393+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T19:00:52.543393+0000","last_undegraded":"2023-12-24T19:00:52.543393+0000","last_fullsized":"2023-12-24T19:00:52.543393+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10732,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543086+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T19:00:52.543086+0000","last_peered":"2023-12-24T19:00:52.543086+0000","last_clean":"2023-12-24T19:00:52.543086+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T19:00:52.543086+0000","last_undegraded":"2023-12-24T19:00:52.543086+0000","last_fullsized":"2023-12-24T19:00:52.543086+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3508,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:04:46.246536+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:04:46.246536+0000","last_peered":"2023-12-24T21:04:46.246536+0000","last_clean":"2023-12-24T21:04:46.246536+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:04:46.246536+0000","last_undegraded":"2023-12-24T21:04:46.246536+0000","last_fullsized":"2023-12-24T21:04:46.246536+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 103s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12811,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:05.282556+0000","last_change":"2023-12-24T21:08:05.282556+0000","last_active":"2023-12-24T21:08:05.282556+0000","last_peered":"2023-12-24T21:08:05.282556+0000","last_clean":"2023-12-24T21:08:05.282556+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:08:05.282556+0000","last_undegraded":"2023-12-24T21:08:05.282556+0000","last_fullsized":"2023-12-24T21:08:05.282556+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3084,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:07:55.330762+0000","last_change":"2023-12-24T21:07:55.330762+0000","last_active":"2023-12-24T21:07:55.330762+0000","last_peered":"2023-12-24T21:07:55.330762+0000","last_clean":"2023-12-24T21:07:55.330762+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:07:55.330762+0000","last_undegraded":"2023-12-24T21:07:55.330762+0000","last_fullsized":"2023-12-24T21:07:55.330762+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3314,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543485+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T19:00:52.543485+0000","last_peered":"2023-12-24T19:00:52.543485+0000","last_clean":"2023-12-24T19:00:52.543485+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T19:00:52.543485+0000","last_undegraded":"2023-12-24T19:00:52.543485+0000","last_fullsized":"2023-12-24T19:00:52.543485+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740661,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,4,6,10,6],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:07:59 2023","interfaces":[{"interface":"back","average":{"1min":0.254,"5min":0.221,"15min":0.217},"min":{"1min":0.153,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.23400000000000001,"5min":0.24099999999999999,"15min":0.23999999999999999},"min":{"1min":0.184,"5min":0.17499999999999999,"15min":0.16500000000000001},"max":{"1min":0.44800000000000001,"5min":0.45600000000000002,"15min":0.48699999999999999},"last":0.24099999999999999}]},{"osd":1,"last update":"Sun Dec 24 21:07:59 2023","interfaces":[{"interface":"back","average":{"1min":0.23699999999999999,"5min":0.24299999999999999,"15min":0.23599999999999999},"min":{"1min":0.19800000000000001,"5min":0.19,"15min":0.17299999999999999},"max":{"1min":0.434,"5min":0.44900000000000001,"15min":0.46300000000000002},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.255,"5min":0.26200000000000001,"15min":0.26100000000000001},"min":{"1min":0.19900000000000001,"5min":0.191,"15min":0.17799999999999999},"max":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"last":0.31900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359740660,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:08:06 2023","interfaces":[{"interface":"back","average":{"1min":0.184,"5min":0.188,"15min":0.19900000000000001},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.13},"max":{"1min":0.28599999999999998,"5min":0.41099999999999998,"15min":0.42699999999999999},"last":0.17999999999999999},{"interface":"front","average":{"1min":0.29799999999999999,"5min":0.23899999999999999,"15min":0.23200000000000001},"min":{"1min":0.14799999999999999,"5min":0.14799999999999999,"15min":0.14499999999999999},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.27600000000000002}]},{"osd":2,"last update":"Sun Dec 24 21:08:06 2023","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.25900000000000001,"15min":0.253},"min":{"1min":0.17699999999999999,"5min":0.17100000000000001,"15min":0.16800000000000001},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.314},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.28999999999999998,"15min":0.28199999999999997},"min":{"1min":0.22,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.33900000000000002}]}]},{"osd":1,"up_from":8,"seq":34359740663,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:08:08 2023","interfaces":[{"interface":"back","average":{"1min":0.20100000000000001,"5min":0.19600000000000001,"15min":0.20000000000000001},"min":{"1min":0.14399999999999999,"5min":0.126,"15min":0.126},"max":{"1min":0.36699999999999999,"5min":0.437,"15min":0.437},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.23200000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"min":{"1min":0.183,"5min":0.153,"15min":0.153},"max":{"1min":0.39600000000000002,"5min":0.439,"15min":0.45800000000000002},"last":0.27300000000000002}]},{"osd":2,"last update":"Sun Dec 24 21:08:08 2023","interfaces":[{"interface":"back","average":{"1min":0.23000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"min":{"1min":0.17399999999999999,"5min":0.157,"15min":0.157},"max":{"1min":0.379,"5min":0.45600000000000002,"15min":0.46999999999999997},"last":0.22},{"interface":"front","average":{"1min":0.27400000000000002,"5min":0.26900000000000002,"15min":0.26500000000000001},"min":{"1min":0.21199999999999999,"5min":0.192,"15min":0.16900000000000001},"max":{"1min":0.42699999999999999,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.30099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:08:12.975 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:12.975 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:12.975 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:12.975 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:12.976 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:12.976 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:12.976 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:12.976 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:12.976 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:12.977 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:12.977 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:12.977 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:12.977 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:12.978 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:08:13.188 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:13.193+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:13.228 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:13.233+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:13.524 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:13.529+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:14.198 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:14.201+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:14.266 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:14.271+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:14.552 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:14.557+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:15.156 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:15.161+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:15.226 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:15.231+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:15.524 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:15.530+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:16.184 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:16.189+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:16.205 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:16.210+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:16.517 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:16.523+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:17.212 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:17.217+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:17.250 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:17.255+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:17.505 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:17.511+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:18.187 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:18.192+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:18.221 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:18.226+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:18.552 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:18.557+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:19.196 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:19.201+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:19.207 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:19.212+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:19.547 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:19.552+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:20.176 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:20.181+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:20.202 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:20.207+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:20.536 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:20.541+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:21.163 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:21.168+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:21.174 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:21.179+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:21.500 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:21.505+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:22.180 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:22.186+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:22.204 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:22.210+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:22.534 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:22.540+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:23.169 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:23.175+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:23.210 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:23.216+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:23.537 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:23.543+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:24.179 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:24.185+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:24.186 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:24.193+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:24.587 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:24.593+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:25.137 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:25.143+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:25.188 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:25.195+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:25.612 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:25.618+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:26.109 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:26.115+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:26.239 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:26.245+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:26.638 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:26.644+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:27.090 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:27.097+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:27.254 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:27.260+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:27.676 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:27.682+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:28.072 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:28.078+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:28.279 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:28.285+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:28.665 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:28.671+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:29.050 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:29.056+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:29.303 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:29.309+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:29.713 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:29.719+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:30.099 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:30.104+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:30.268 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:30.274+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:30.701 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:30.707+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:31.141 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:31.147+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:31.251 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:31.257+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:31.671 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:31.677+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:32.190 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:32.196+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:32.274 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:32.280+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:32.713 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:32.719+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:32.979 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:08:33.233 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:33.239+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:33.234 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:33.240+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:33.315 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:08:33.317 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:08:33.328 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":5885,"stamp":"2023-12-24T21:08:32.647931+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,4,6,10,518],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001652"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2825,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543752+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T19:00:52.543752+0000","last_peered":"2023-12-24T19:00:52.543752+0000","last_clean":"2023-12-24T19:00:52.543752+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T19:00:52.543752+0000","last_undegraded":"2023-12-24T19:00:52.543752+0000","last_fullsized":"2023-12-24T19:00:52.543752+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3619,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543664+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T19:00:52.543664+0000","last_peered":"2023-12-24T19:00:52.543664+0000","last_clean":"2023-12-24T19:00:52.543664+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T19:00:52.543664+0000","last_undegraded":"2023-12-24T19:00:52.543664+0000","last_fullsized":"2023-12-24T19:00:52.543664+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2982,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543574+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T19:00:52.543574+0000","last_peered":"2023-12-24T19:00:52.543574+0000","last_clean":"2023-12-24T19:00:52.543574+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T19:00:52.543574+0000","last_undegraded":"2023-12-24T19:00:52.543574+0000","last_fullsized":"2023-12-24T19:00:52.543574+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10109,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:31.257051+0000","last_change":"2023-12-24T21:08:31.257051+0000","last_active":"2023-12-24T21:08:31.257051+0000","last_peered":"2023-12-24T21:08:31.257051+0000","last_clean":"2023-12-24T21:08:31.257051+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:08:31.257051+0000","last_undegraded":"2023-12-24T21:08:31.257051+0000","last_fullsized":"2023-12-24T21:08:31.257051+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3924,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:04:44.168071+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:04:44.168071+0000","last_peered":"2023-12-24T21:04:44.168071+0000","last_clean":"2023-12-24T21:04:44.168071+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:04:44.168071+0000","last_undegraded":"2023-12-24T21:04:44.168071+0000","last_fullsized":"2023-12-24T21:04:44.168071+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11204,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543302+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T19:00:52.543302+0000","last_peered":"2023-12-24T19:00:52.543302+0000","last_clean":"2023-12-24T19:00:52.543302+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T19:00:52.543302+0000","last_undegraded":"2023-12-24T19:00:52.543302+0000","last_fullsized":"2023-12-24T19:00:52.543302+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":285,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543005+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T19:00:52.543005+0000","last_peered":"2023-12-24T19:00:52.543005+0000","last_clean":"2023-12-24T19:00:52.543005+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T19:00:52.543005+0000","last_undegraded":"2023-12-24T19:00:52.543005+0000","last_fullsized":"2023-12-24T19:00:52.543005+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3530,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543393+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T19:00:52.543393+0000","last_peered":"2023-12-24T19:00:52.543393+0000","last_clean":"2023-12-24T19:00:52.543393+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T19:00:52.543393+0000","last_undegraded":"2023-12-24T19:00:52.543393+0000","last_fullsized":"2023-12-24T19:00:52.543393+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10732,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543086+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T19:00:52.543086+0000","last_peered":"2023-12-24T19:00:52.543086+0000","last_clean":"2023-12-24T19:00:52.543086+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T19:00:52.543086+0000","last_undegraded":"2023-12-24T19:00:52.543086+0000","last_fullsized":"2023-12-24T19:00:52.543086+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3508,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:04:46.246536+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:04:46.246536+0000","last_peered":"2023-12-24T21:04:46.246536+0000","last_clean":"2023-12-24T21:04:46.246536+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:04:46.246536+0000","last_undegraded":"2023-12-24T21:04:46.246536+0000","last_fullsized":"2023-12-24T21:04:46.246536+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 103s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12814,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:21.180274+0000","last_change":"2023-12-24T21:08:21.180274+0000","last_active":"2023-12-24T21:08:21.180274+0000","last_peered":"2023-12-24T21:08:21.180274+0000","last_clean":"2023-12-24T21:08:21.180274+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:08:21.180274+0000","last_undegraded":"2023-12-24T21:08:21.180274+0000","last_fullsized":"2023-12-24T21:08:21.180274+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3090,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:27.259857+0000","last_change":"2023-12-24T21:08:27.259857+0000","last_active":"2023-12-24T21:08:27.259857+0000","last_peered":"2023-12-24T21:08:27.259857+0000","last_clean":"2023-12-24T21:08:27.259857+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:08:27.259857+0000","last_undegraded":"2023-12-24T21:08:27.259857+0000","last_fullsized":"2023-12-24T21:08:27.259857+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3314,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T19:00:52.543485+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T19:00:52.543485+0000","last_peered":"2023-12-24T19:00:52.543485+0000","last_clean":"2023-12-24T19:00:52.543485+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T19:00:52.543485+0000","last_undegraded":"2023-12-24T19:00:52.543485+0000","last_fullsized":"2023-12-24T19:00:52.543485+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740665,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,4,6,10,6],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:07:59 2023","interfaces":[{"interface":"back","average":{"1min":0.254,"5min":0.221,"15min":0.217},"min":{"1min":0.153,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.16700000000000001},{"interface":"front","average":{"1min":0.23400000000000001,"5min":0.24099999999999999,"15min":0.23999999999999999},"min":{"1min":0.184,"5min":0.17499999999999999,"15min":0.16500000000000001},"max":{"1min":0.44800000000000001,"5min":0.45600000000000002,"15min":0.48699999999999999},"last":0.19900000000000001}]},{"osd":1,"last update":"Sun Dec 24 21:07:59 2023","interfaces":[{"interface":"back","average":{"1min":0.23699999999999999,"5min":0.24299999999999999,"15min":0.23599999999999999},"min":{"1min":0.19800000000000001,"5min":0.19,"15min":0.17299999999999999},"max":{"1min":0.434,"5min":0.44900000000000001,"15min":0.46300000000000002},"last":0.187},{"interface":"front","average":{"1min":0.255,"5min":0.26200000000000001,"15min":0.26100000000000001},"min":{"1min":0.19900000000000001,"5min":0.191,"15min":0.17799999999999999},"max":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"last":0.22900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359740664,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:08:06 2023","interfaces":[{"interface":"back","average":{"1min":0.184,"5min":0.188,"15min":0.19900000000000001},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.13},"max":{"1min":0.28599999999999998,"5min":0.41099999999999998,"15min":0.42699999999999999},"last":0.186},{"interface":"front","average":{"1min":0.29799999999999999,"5min":0.23899999999999999,"15min":0.23200000000000001},"min":{"1min":0.14799999999999999,"5min":0.14799999999999999,"15min":0.14499999999999999},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.23899999999999999}]},{"osd":2,"last update":"Sun Dec 24 21:08:06 2023","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.25900000000000001,"15min":0.253},"min":{"1min":0.17699999999999999,"5min":0.17100000000000001,"15min":0.16800000000000001},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.28999999999999998,"15min":0.28199999999999997},"min":{"1min":0.22,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.26400000000000001}]}]},{"osd":1,"up_from":8,"seq":34359740667,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:08:08 2023","interfaces":[{"interface":"back","average":{"1min":0.20100000000000001,"5min":0.19600000000000001,"15min":0.20000000000000001},"min":{"1min":0.14399999999999999,"5min":0.126,"15min":0.126},"max":{"1min":0.36699999999999999,"5min":0.437,"15min":0.437},"last":0.224},{"interface":"front","average":{"1min":0.23200000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"min":{"1min":0.183,"5min":0.153,"15min":0.153},"max":{"1min":0.39600000000000002,"5min":0.439,"15min":0.45800000000000002},"last":0.26900000000000002}]},{"osd":2,"last update":"Sun Dec 24 21:08:08 2023","interfaces":[{"interface":"back","average":{"1min":0.23000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"min":{"1min":0.17399999999999999,"5min":0.157,"15min":0.157},"max":{"1min":0.379,"5min":0.45600000000000002,"15min":0.46999999999999997},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.27400000000000002,"5min":0.26900000000000002,"15min":0.26500000000000001},"min":{"1min":0.21199999999999999,"5min":0.192,"15min":0.16900000000000001},"max":{"1min":0.42699999999999999,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.30399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:08:33.329 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:33.329 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:33.330 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:33.330 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:33.330 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:33.330 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:33.330 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:33.330 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:33.331 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:33.331 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:33.331 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:33.331 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:33.331 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:33.331 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.7 2023-12-24T21:08:33.671 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.7 on osd.1 to deep-scrub 2023-12-24T21:08:33.683 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.6 2023-12-24T21:08:33.727 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:33.733+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:34.020 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.6 on osd.1 to deep-scrub 2023-12-24T21:08:34.031 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2023-12-24T21:08:34.243 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:34.249+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:34.268 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:34.274+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:34.388 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2023-12-24T21:08:34.398 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-12-24T21:08:34.682 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:34.688+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:34.755 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-12-24T21:08:34.763 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.0 2023-12-24T21:08:35.131 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.0 on osd.2 to deep-scrub 2023-12-24T21:08:35.143 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-12-24T21:08:35.205 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:35.210+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:35.252 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:35.256+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:35.502 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-12-24T21:08:35.513 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.0 2023-12-24T21:08:35.670 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:35.675+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:35.840 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 1.0 on osd.1 to deep-scrub 2023-12-24T21:08:35.850 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.3 2023-12-24T21:08:36.185 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:36.190+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:36.191 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.3 on osd.1 to deep-scrub 2023-12-24T21:08:36.203 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-12-24T21:08:36.265 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:36.270+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:36.542 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-12-24T21:08:36.552 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.1 2023-12-24T21:08:36.715 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:36.720+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:36.925 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.1 on osd.2 to deep-scrub 2023-12-24T21:08:36.937 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-12-24T21:08:37.192 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:37.197+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:37.250 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:37.255+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:37.265 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-12-24T21:08:37.295 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.2 2023-12-24T21:08:37.631 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.2 on osd.0 to deep-scrub 2023-12-24T21:08:37.643 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.4 2023-12-24T21:08:37.755 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:37.760+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:37.974 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.4 on osd.1 to deep-scrub 2023-12-24T21:08:37.986 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:08:38.185 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:38.190+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:38.277 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:38.282+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:38.796 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:38.801+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:39.162 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:39.167+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:39.309 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:39.314+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:39.761 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:39.766+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:40.194 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:40.199+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:40.347 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:40.352+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:40.726 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:40.731+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:41.165 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:41.170+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:41.373 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:41.378+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:41.724 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:41.729+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:42.165 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:42.170+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:42.406 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:42.411+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:42.767 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:42.772+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:43.177 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:43.182+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:43.395 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:43.400+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:43.805 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:43.810+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:44.175 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:44.180+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:44.433 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:44.438+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:44.850 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:44.855+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:45.153 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:45.158+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:45.454 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:45.458+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:45.894 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:45.899+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:46.134 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:46.139+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:46.477 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:46.482+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:46.905 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:46.910+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:47.122 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:47.127+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:47.511 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:47.516+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:47.924 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:47.929+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:48.083 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:48.088+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:48.489 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:48.494+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:48.930 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:48.934+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:49.054 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:49.059+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:49.537 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:49.541+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:49.917 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:49.922+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:50.071 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:50.076+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:50.565 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:50.570+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:50.956 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:50.961+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:51.098 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:51.102+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:51.610 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:51.614+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:51.910 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:51.915+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:52.109 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:52.114+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:52.627 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:52.631+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:52.885 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:52.890+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:53.102 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:53.107+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:53.646 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:53.651+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:53.869 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:53.873+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:54.142 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:54.146+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:54.675 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:54.680+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:54.855 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:54.860+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:55.184 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:55.189+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:55.694 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:55.698+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:55.857 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:55.862+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:56.166 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:56.171+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:56.689 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:56.694+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:56.873 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:56.878+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:57.133 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:57.137+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:57.662 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:57.667+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:57.838 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:57.843+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:57.987 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:08:58.156 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:58.161+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:58.352 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:08:58.354 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:08:58.366 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":5897,"stamp":"2023-12-24T21:08:56.651525+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,4,6,10,518],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001894"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2826,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:26.705740+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T21:06:26.705740+0000","last_peered":"2023-12-24T21:06:26.705740+0000","last_clean":"2023-12-24T21:06:26.705740+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T21:06:26.705740+0000","last_undegraded":"2023-12-24T21:06:26.705740+0000","last_fullsized":"2023-12-24T21:06:26.705740+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3620,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:27.063375+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T21:06:27.063375+0000","last_peered":"2023-12-24T21:06:27.063375+0000","last_clean":"2023-12-24T21:06:27.063375+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T21:06:27.063375+0000","last_undegraded":"2023-12-24T21:06:27.063375+0000","last_fullsized":"2023-12-24T21:06:27.063375+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2983,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:27.404638+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T21:06:27.404638+0000","last_peered":"2023-12-24T21:06:27.404638+0000","last_clean":"2023-12-24T21:06:27.404638+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T21:06:27.404638+0000","last_undegraded":"2023-12-24T21:06:27.404638+0000","last_fullsized":"2023-12-24T21:06:27.404638+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10116,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:43.401508+0000","last_change":"2023-12-24T21:08:43.401508+0000","last_active":"2023-12-24T21:08:43.401508+0000","last_peered":"2023-12-24T21:08:43.401508+0000","last_clean":"2023-12-24T21:08:43.401508+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:08:43.401508+0000","last_undegraded":"2023-12-24T21:08:43.401508+0000","last_fullsized":"2023-12-24T21:08:43.401508+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3925,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:28.157180+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:06:28.157180+0000","last_peered":"2023-12-24T21:06:28.157180+0000","last_clean":"2023-12-24T21:06:28.157180+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:06:28.157180+0000","last_undegraded":"2023-12-24T21:06:28.157180+0000","last_fullsized":"2023-12-24T21:06:28.157180+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11205,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:28.497188+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T21:06:28.497188+0000","last_peered":"2023-12-24T21:06:28.497188+0000","last_clean":"2023-12-24T21:06:28.497188+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T21:06:28.497188+0000","last_undegraded":"2023-12-24T21:06:28.497188+0000","last_fullsized":"2023-12-24T21:06:28.497188+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":286,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:28.874717+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T21:06:28.874717+0000","last_peered":"2023-12-24T21:06:28.874717+0000","last_clean":"2023-12-24T21:06:28.874717+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T21:06:28.874717+0000","last_undegraded":"2023-12-24T21:06:28.874717+0000","last_fullsized":"2023-12-24T21:06:28.874717+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3531,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:29.208984+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T21:06:29.208984+0000","last_peered":"2023-12-24T21:06:29.208984+0000","last_clean":"2023-12-24T21:06:29.208984+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T21:06:29.208984+0000","last_undegraded":"2023-12-24T21:06:29.208984+0000","last_fullsized":"2023-12-24T21:06:29.208984+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10733,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:29.598630+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T21:06:29.598630+0000","last_peered":"2023-12-24T21:06:29.598630+0000","last_clean":"2023-12-24T21:06:29.598630+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T21:06:29.598630+0000","last_undegraded":"2023-12-24T21:06:29.598630+0000","last_fullsized":"2023-12-24T21:06:29.598630+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3509,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:06:29.987704+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:06:29.987704+0000","last_peered":"2023-12-24T21:06:29.987704+0000","last_clean":"2023-12-24T21:06:29.987704+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:06:29.987704+0000","last_undegraded":"2023-12-24T21:06:29.987704+0000","last_fullsized":"2023-12-24T21:06:29.987704+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 230s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12824,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:48.495462+0000","last_change":"2023-12-24T21:08:48.495462+0000","last_active":"2023-12-24T21:08:48.495462+0000","last_peered":"2023-12-24T21:08:48.495462+0000","last_clean":"2023-12-24T21:08:48.495462+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:08:48.495462+0000","last_undegraded":"2023-12-24T21:08:48.495462+0000","last_fullsized":"2023-12-24T21:08:48.495462+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3097,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:40.353260+0000","last_change":"2023-12-24T21:08:40.353260+0000","last_active":"2023-12-24T21:08:40.353260+0000","last_peered":"2023-12-24T21:08:40.353260+0000","last_clean":"2023-12-24T21:08:40.353260+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:08:40.353260+0000","last_undegraded":"2023-12-24T21:08:40.353260+0000","last_fullsized":"2023-12-24T21:08:40.353260+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3315,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:31.122557+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T21:06:31.122557+0000","last_peered":"2023-12-24T21:06:31.122557+0000","last_clean":"2023-12-24T21:06:31.122557+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T21:06:31.122557+0000","last_undegraded":"2023-12-24T21:06:31.122557+0000","last_fullsized":"2023-12-24T21:06:31.122557+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740669,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,4,6,10,6],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:07:59 2023","interfaces":[{"interface":"back","average":{"1min":0.254,"5min":0.221,"15min":0.217},"min":{"1min":0.153,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.23400000000000001,"5min":0.24099999999999999,"15min":0.23999999999999999},"min":{"1min":0.184,"5min":0.17499999999999999,"15min":0.16500000000000001},"max":{"1min":0.44800000000000001,"5min":0.45600000000000002,"15min":0.48699999999999999},"last":0.23999999999999999}]},{"osd":1,"last update":"Sun Dec 24 21:07:59 2023","interfaces":[{"interface":"back","average":{"1min":0.23699999999999999,"5min":0.24299999999999999,"15min":0.23599999999999999},"min":{"1min":0.19800000000000001,"5min":0.19,"15min":0.17299999999999999},"max":{"1min":0.434,"5min":0.44900000000000001,"15min":0.46300000000000002},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.255,"5min":0.26200000000000001,"15min":0.26100000000000001},"min":{"1min":0.19900000000000001,"5min":0.191,"15min":0.17799999999999999},"max":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"last":0.316}]}]},{"osd":0,"up_from":8,"seq":34359740668,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:08:06 2023","interfaces":[{"interface":"back","average":{"1min":0.184,"5min":0.188,"15min":0.19900000000000001},"min":{"1min":0.151,"5min":0.14299999999999999,"15min":0.13},"max":{"1min":0.28599999999999998,"5min":0.41099999999999998,"15min":0.42699999999999999},"last":0.216},{"interface":"front","average":{"1min":0.29799999999999999,"5min":0.23899999999999999,"15min":0.23200000000000001},"min":{"1min":0.14799999999999999,"5min":0.14799999999999999,"15min":0.14499999999999999},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.246}]},{"osd":2,"last update":"Sun Dec 24 21:08:06 2023","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.25900000000000001,"15min":0.253},"min":{"1min":0.17699999999999999,"5min":0.17100000000000001,"15min":0.16800000000000001},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.28999999999999998,"15min":0.28199999999999997},"min":{"1min":0.22,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.309}]}]},{"osd":1,"up_from":8,"seq":34359740671,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:08:08 2023","interfaces":[{"interface":"back","average":{"1min":0.20100000000000001,"5min":0.19600000000000001,"15min":0.20000000000000001},"min":{"1min":0.14399999999999999,"5min":0.126,"15min":0.126},"max":{"1min":0.36699999999999999,"5min":0.437,"15min":0.437},"last":0.17899999999999999},{"interface":"front","average":{"1min":0.23200000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"min":{"1min":0.183,"5min":0.153,"15min":0.153},"max":{"1min":0.39600000000000002,"5min":0.439,"15min":0.45800000000000002},"last":0.216}]},{"osd":2,"last update":"Sun Dec 24 21:08:08 2023","interfaces":[{"interface":"back","average":{"1min":0.23000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"min":{"1min":0.17399999999999999,"5min":0.157,"15min":0.157},"max":{"1min":0.379,"5min":0.45600000000000002,"15min":0.46999999999999997},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.27400000000000002,"5min":0.26900000000000002,"15min":0.26500000000000001},"min":{"1min":0.21199999999999999,"5min":0.192,"15min":0.16900000000000001},"max":{"1min":0.42699999999999999,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.28799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:08:58.367 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:58.367 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:58.367 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:58.367 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:58.368 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:58.368 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:58.368 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:58.368 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:58.368 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:58.368 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:58.368 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:58.369 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:58.369 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:08:58.369 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:08:58.662 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:58.666+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:58.820 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:58.824+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:59.120 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:08:59.124+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:59.616 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:08:59.621+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:08:59.793 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:08:59.798+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:00.142 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:00.146+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:00.649 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:00.653+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:00.830 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:00.835+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:01.190 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:01.195+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:01.639 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:01.644+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:01.805 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:01.809+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:02.167 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:02.171+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:02.653 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:02.658+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:02.851 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:02.856+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:03.216 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:03.221+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:03.657 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:03.661+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:03.883 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:03.888+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:04.218 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:04.223+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:04.687 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:04.692+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:04.926 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:04.930+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:05.258 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:05.263+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:05.648 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:05.652+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:05.975 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:05.979+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:06.257 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:06.263+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:06.663 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:06.668+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:07.007 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:07.012+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:07.276 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:07.282+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:07.663 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:07.669+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:08.037 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:08.043+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:08.246 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:08.252+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:08.645 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:08.650+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:09.087 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:09.092+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:09.262 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:09.268+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:09.610 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:09.615+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:10.064 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:10.070+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:10.253 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:10.258+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:10.564 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:10.569+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:11.036 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:11.042+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:11.303 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:11.308+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:11.576 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:11.582+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:12.015 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:12.020+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:12.309 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:12.314+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:12.541 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:12.546+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:13.053 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:13.058+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:13.356 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:13.361+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:13.520 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:13.525+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:14.023 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:14.028+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:14.355 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:14.361+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:14.534 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:14.539+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:14.975 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:14.980+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:15.387 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:15.392+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:15.499 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:15.505+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:15.934 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:15.939+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:16.354 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:16.359+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:16.465 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:16.470+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:16.891 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:16.897+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:17.327 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:17.332+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:17.428 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:17.433+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:17.917 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:17.922+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:18.365 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:18.371+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:18.370 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:09:18.417 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:18.422+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:18.736 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:09:18.739 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:09:18.751 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":5908,"stamp":"2023-12-24T21:09:18.654966+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,4,6,10,518],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001646"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2826,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:26.705740+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T21:06:26.705740+0000","last_peered":"2023-12-24T21:06:26.705740+0000","last_clean":"2023-12-24T21:06:26.705740+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T21:06:26.705740+0000","last_undegraded":"2023-12-24T21:06:26.705740+0000","last_fullsized":"2023-12-24T21:06:26.705740+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3620,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:27.063375+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T21:06:27.063375+0000","last_peered":"2023-12-24T21:06:27.063375+0000","last_clean":"2023-12-24T21:06:27.063375+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T21:06:27.063375+0000","last_undegraded":"2023-12-24T21:06:27.063375+0000","last_fullsized":"2023-12-24T21:06:27.063375+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2983,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:27.404638+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T21:06:27.404638+0000","last_peered":"2023-12-24T21:06:27.404638+0000","last_clean":"2023-12-24T21:06:27.404638+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T21:06:27.404638+0000","last_undegraded":"2023-12-24T21:06:27.404638+0000","last_fullsized":"2023-12-24T21:06:27.404638+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10122,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:09:09.615856+0000","last_change":"2023-12-24T21:09:09.615856+0000","last_active":"2023-12-24T21:09:09.615856+0000","last_peered":"2023-12-24T21:09:09.615856+0000","last_clean":"2023-12-24T21:09:09.615856+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:09:09.615856+0000","last_undegraded":"2023-12-24T21:09:09.615856+0000","last_fullsized":"2023-12-24T21:09:09.615856+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3925,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:28.157180+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:06:28.157180+0000","last_peered":"2023-12-24T21:06:28.157180+0000","last_clean":"2023-12-24T21:06:28.157180+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:06:28.157180+0000","last_undegraded":"2023-12-24T21:06:28.157180+0000","last_fullsized":"2023-12-24T21:06:28.157180+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11205,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:28.497188+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T21:06:28.497188+0000","last_peered":"2023-12-24T21:06:28.497188+0000","last_clean":"2023-12-24T21:06:28.497188+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T21:06:28.497188+0000","last_undegraded":"2023-12-24T21:06:28.497188+0000","last_fullsized":"2023-12-24T21:06:28.497188+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":286,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:28.874717+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T21:06:28.874717+0000","last_peered":"2023-12-24T21:06:28.874717+0000","last_clean":"2023-12-24T21:06:28.874717+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T21:06:28.874717+0000","last_undegraded":"2023-12-24T21:06:28.874717+0000","last_fullsized":"2023-12-24T21:06:28.874717+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3531,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:29.208984+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T21:06:29.208984+0000","last_peered":"2023-12-24T21:06:29.208984+0000","last_clean":"2023-12-24T21:06:29.208984+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T21:06:29.208984+0000","last_undegraded":"2023-12-24T21:06:29.208984+0000","last_fullsized":"2023-12-24T21:06:29.208984+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10733,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:29.598630+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T21:06:29.598630+0000","last_peered":"2023-12-24T21:06:29.598630+0000","last_clean":"2023-12-24T21:06:29.598630+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T21:06:29.598630+0000","last_undegraded":"2023-12-24T21:06:29.598630+0000","last_fullsized":"2023-12-24T21:06:29.598630+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3509,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:06:29.987704+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:06:29.987704+0000","last_peered":"2023-12-24T21:06:29.987704+0000","last_clean":"2023-12-24T21:06:29.987704+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:06:29.987704+0000","last_undegraded":"2023-12-24T21:06:29.987704+0000","last_fullsized":"2023-12-24T21:06:29.987704+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 230s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12830,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:09:13.525668+0000","last_change":"2023-12-24T21:09:13.525668+0000","last_active":"2023-12-24T21:09:13.525668+0000","last_peered":"2023-12-24T21:09:13.525668+0000","last_clean":"2023-12-24T21:09:13.525668+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:09:13.525668+0000","last_undegraded":"2023-12-24T21:09:13.525668+0000","last_fullsized":"2023-12-24T21:09:13.525668+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3103,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:09:05.653977+0000","last_change":"2023-12-24T21:09:05.653977+0000","last_active":"2023-12-24T21:09:05.653977+0000","last_peered":"2023-12-24T21:09:05.653977+0000","last_clean":"2023-12-24T21:09:05.653977+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:09:05.653977+0000","last_undegraded":"2023-12-24T21:09:05.653977+0000","last_fullsized":"2023-12-24T21:09:05.653977+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3315,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:31.122557+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T21:06:31.122557+0000","last_peered":"2023-12-24T21:06:31.122557+0000","last_clean":"2023-12-24T21:06:31.122557+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T21:06:31.122557+0000","last_undegraded":"2023-12-24T21:06:31.122557+0000","last_fullsized":"2023-12-24T21:06:31.122557+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740674,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,4,6,10,6],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:09:02 2023","interfaces":[{"interface":"back","average":{"1min":0.23699999999999999,"5min":0.22800000000000001,"15min":0.219},"min":{"1min":0.159,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":1.0229999999999999,"5min":1.198,"15min":1.198},"last":0.19900000000000001},{"interface":"front","average":{"1min":0.26400000000000001,"5min":0.248,"15min":0.24199999999999999},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16500000000000001},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Sun Dec 24 21:09:02 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.254,"15min":0.23999999999999999},"min":{"1min":0.187,"5min":0.187,"15min":0.17299999999999999},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.29299999999999998,"5min":0.26900000000000002,"15min":0.26300000000000001},"min":{"1min":0.19900000000000001,"5min":0.191,"15min":0.17799999999999999},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.25900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359740673,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:09:08 2023","interfaces":[{"interface":"back","average":{"1min":0.186,"5min":0.189,"15min":0.19700000000000001},"min":{"1min":0.154,"5min":0.14999999999999999,"15min":0.13},"max":{"1min":0.22800000000000001,"5min":0.41099999999999998,"15min":0.42699999999999999},"last":0.17199999999999999},{"interface":"front","average":{"1min":0.253,"5min":0.246,"15min":0.23300000000000001},"min":{"1min":0.189,"5min":0.14799999999999999,"15min":0.14499999999999999},"max":{"1min":0.55900000000000005,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.19700000000000001}]},{"osd":2,"last update":"Sun Dec 24 21:09:08 2023","interfaces":[{"interface":"back","average":{"1min":0.26800000000000002,"5min":0.26300000000000001,"15min":0.254},"min":{"1min":0.19800000000000001,"5min":0.17100000000000001,"15min":0.16800000000000001},"max":{"1min":0.58699999999999997,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.29799999999999999,"5min":0.29399999999999998,"15min":0.28199999999999997},"min":{"1min":0.224,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":0.60299999999999998,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.33300000000000002}]}]},{"osd":1,"up_from":8,"seq":34359740676,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:09:10 2023","interfaces":[{"interface":"back","average":{"1min":0.20100000000000001,"5min":0.19800000000000001,"15min":0.20000000000000001},"min":{"1min":0.16200000000000001,"5min":0.126,"15min":0.126},"max":{"1min":0.23899999999999999,"5min":0.437,"15min":0.437},"last":0.20100000000000001},{"interface":"front","average":{"1min":0.22800000000000001,"5min":0.22900000000000001,"15min":0.22800000000000001},"min":{"1min":0.193,"5min":0.153,"15min":0.153},"max":{"1min":0.29199999999999998,"5min":0.439,"15min":0.45800000000000002},"last":0.22700000000000001}]},{"osd":2,"last update":"Sun Dec 24 21:09:10 2023","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":0.23599999999999999,"15min":0.24099999999999999},"min":{"1min":0.184,"5min":0.157,"15min":0.157},"max":{"1min":0.28399999999999997,"5min":0.45600000000000002,"15min":0.46999999999999997},"last":0.19},{"interface":"front","average":{"1min":0.27500000000000002,"5min":0.27100000000000002,"15min":0.26600000000000001},"min":{"1min":0.23400000000000001,"5min":0.192,"15min":0.16900000000000001},"max":{"1min":0.33500000000000002,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.249}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:09:18.752 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:18.753 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:18.753 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:18.753 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:18.753 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:18.753 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:18.754 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:18.754 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:18.754 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:18.754 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:18.754 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:18.754 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:18.755 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:18.755 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:09:18.932 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:18.938+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:19.362 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:19.367+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:19.373 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:19.378+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:19.978 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:19.983+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:20.325 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:20.329+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:20.388 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:20.392+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:21.008 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:21.013+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:21.338 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:21.342+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:21.339 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:21.343+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:21.969 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:21.973+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:22.376 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:22.380+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:22.383 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:22.387+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:22.988 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:22.993+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:23.333 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:23.337+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:23.347 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:23.351+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:23.969 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:23.973+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:24.347 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:24.351+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:24.382 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:24.387+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:24.948 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:24.953+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:25.381 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:25.385+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:25.411 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:25.415+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:25.971 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:25.976+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:26.407 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:26.411+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:26.418 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:26.423+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:26.955 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:26.960+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:27.409 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:27.413+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:27.450 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:27.454+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:27.946 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:27.950+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:28.421 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:28.425+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:28.447 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:28.451+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:28.923 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:28.927+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:29.375 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:29.380+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:29.425 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:29.429+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:29.892 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:29.896+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:30.420 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:30.424+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:30.452 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:30.457+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:30.912 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:30.916+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:31.380 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:31.384+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:31.488 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:31.493+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:31.896 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:31.900+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:32.382 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:32.386+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:32.442 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:32.447+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:32.894 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:32.899+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:33.389 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:33.393+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:33.429 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:33.433+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:33.933 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:33.937+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:34.412 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:34.416+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:34.455 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:34.459+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:34.953 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:34.957+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:35.424 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:35.428+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:35.464 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:35.468+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:35.915 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:35.919+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:36.405 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:36.409+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:36.438 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:36.442+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:36.884 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:36.888+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:37.417 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:37.421+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:37.422 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:37.426+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:37.867 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:37.871+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:38.415 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:38.419+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:38.462 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:38.466+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:38.755 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:09:38.838 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:38.842+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:39.130 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:09:39.131 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:09:39.143 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":5918,"stamp":"2023-12-24T21:09:38.658109+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,4,6,10,518],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001884"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2826,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:26.705740+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T21:06:26.705740+0000","last_peered":"2023-12-24T21:06:26.705740+0000","last_clean":"2023-12-24T21:06:26.705740+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T21:06:26.705740+0000","last_undegraded":"2023-12-24T21:06:26.705740+0000","last_fullsized":"2023-12-24T21:06:26.705740+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3620,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:27.063375+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T21:06:27.063375+0000","last_peered":"2023-12-24T21:06:27.063375+0000","last_clean":"2023-12-24T21:06:27.063375+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T21:06:27.063375+0000","last_undegraded":"2023-12-24T21:06:27.063375+0000","last_fullsized":"2023-12-24T21:06:27.063375+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2983,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:27.404638+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T21:06:27.404638+0000","last_peered":"2023-12-24T21:06:27.404638+0000","last_clean":"2023-12-24T21:06:27.404638+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T21:06:27.404638+0000","last_undegraded":"2023-12-24T21:06:27.404638+0000","last_fullsized":"2023-12-24T21:06:27.404638+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10131,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:09:35.469527+0000","last_change":"2023-12-24T21:09:35.469527+0000","last_active":"2023-12-24T21:09:35.469527+0000","last_peered":"2023-12-24T21:09:35.469527+0000","last_clean":"2023-12-24T21:09:35.469527+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:09:35.469527+0000","last_undegraded":"2023-12-24T21:09:35.469527+0000","last_fullsized":"2023-12-24T21:09:35.469527+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3925,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:28.157180+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:06:28.157180+0000","last_peered":"2023-12-24T21:06:28.157180+0000","last_clean":"2023-12-24T21:06:28.157180+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:06:28.157180+0000","last_undegraded":"2023-12-24T21:06:28.157180+0000","last_fullsized":"2023-12-24T21:06:28.157180+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11205,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:28.497188+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T21:06:28.497188+0000","last_peered":"2023-12-24T21:06:28.497188+0000","last_clean":"2023-12-24T21:06:28.497188+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T21:06:28.497188+0000","last_undegraded":"2023-12-24T21:06:28.497188+0000","last_fullsized":"2023-12-24T21:06:28.497188+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":286,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:28.874717+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T21:06:28.874717+0000","last_peered":"2023-12-24T21:06:28.874717+0000","last_clean":"2023-12-24T21:06:28.874717+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T21:06:28.874717+0000","last_undegraded":"2023-12-24T21:06:28.874717+0000","last_fullsized":"2023-12-24T21:06:28.874717+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3531,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:29.208984+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T21:06:29.208984+0000","last_peered":"2023-12-24T21:06:29.208984+0000","last_clean":"2023-12-24T21:06:29.208984+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T21:06:29.208984+0000","last_undegraded":"2023-12-24T21:06:29.208984+0000","last_fullsized":"2023-12-24T21:06:29.208984+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10733,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:29.598630+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T21:06:29.598630+0000","last_peered":"2023-12-24T21:06:29.598630+0000","last_clean":"2023-12-24T21:06:29.598630+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T21:06:29.598630+0000","last_undegraded":"2023-12-24T21:06:29.598630+0000","last_fullsized":"2023-12-24T21:06:29.598630+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3509,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:06:29.987704+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:06:29.987704+0000","last_peered":"2023-12-24T21:06:29.987704+0000","last_clean":"2023-12-24T21:06:29.987704+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:06:29.987704+0000","last_undegraded":"2023-12-24T21:06:29.987704+0000","last_fullsized":"2023-12-24T21:06:29.987704+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 230s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12836,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:09:31.493997+0000","last_change":"2023-12-24T21:09:31.493997+0000","last_active":"2023-12-24T21:09:31.493997+0000","last_peered":"2023-12-24T21:09:31.493997+0000","last_clean":"2023-12-24T21:09:31.493997+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:09:31.493997+0000","last_undegraded":"2023-12-24T21:09:31.493997+0000","last_fullsized":"2023-12-24T21:09:31.493997+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3112,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:09:32.447966+0000","last_change":"2023-12-24T21:09:32.447966+0000","last_active":"2023-12-24T21:09:32.447966+0000","last_peered":"2023-12-24T21:09:32.447966+0000","last_clean":"2023-12-24T21:09:32.447966+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:09:32.447966+0000","last_undegraded":"2023-12-24T21:09:32.447966+0000","last_fullsized":"2023-12-24T21:09:32.447966+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3315,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:31.122557+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T21:06:31.122557+0000","last_peered":"2023-12-24T21:06:31.122557+0000","last_clean":"2023-12-24T21:06:31.122557+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T21:06:31.122557+0000","last_undegraded":"2023-12-24T21:06:31.122557+0000","last_fullsized":"2023-12-24T21:06:31.122557+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740678,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,4,6,10,6],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:09:02 2023","interfaces":[{"interface":"back","average":{"1min":0.23699999999999999,"5min":0.22800000000000001,"15min":0.219},"min":{"1min":0.159,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":1.0229999999999999,"5min":1.198,"15min":1.198},"last":0.214},{"interface":"front","average":{"1min":0.26400000000000001,"5min":0.248,"15min":0.24199999999999999},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16500000000000001},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.254}]},{"osd":1,"last update":"Sun Dec 24 21:09:02 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.254,"15min":0.23999999999999999},"min":{"1min":0.187,"5min":0.187,"15min":0.17299999999999999},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.29299999999999998,"5min":0.26900000000000002,"15min":0.26300000000000001},"min":{"1min":0.19900000000000001,"5min":0.191,"15min":0.17799999999999999},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.308}]}]},{"osd":0,"up_from":8,"seq":34359740677,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:09:08 2023","interfaces":[{"interface":"back","average":{"1min":0.186,"5min":0.189,"15min":0.19700000000000001},"min":{"1min":0.154,"5min":0.14999999999999999,"15min":0.13},"max":{"1min":0.22800000000000001,"5min":0.41099999999999998,"15min":0.42699999999999999},"last":0.17000000000000001},{"interface":"front","average":{"1min":0.253,"5min":0.246,"15min":0.23300000000000001},"min":{"1min":0.189,"5min":0.14799999999999999,"15min":0.14499999999999999},"max":{"1min":0.55900000000000005,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.27900000000000003}]},{"osd":2,"last update":"Sun Dec 24 21:09:08 2023","interfaces":[{"interface":"back","average":{"1min":0.26800000000000002,"5min":0.26300000000000001,"15min":0.254},"min":{"1min":0.19800000000000001,"5min":0.17100000000000001,"15min":0.16800000000000001},"max":{"1min":0.58699999999999997,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.246},{"interface":"front","average":{"1min":0.29799999999999999,"5min":0.29399999999999998,"15min":0.28199999999999997},"min":{"1min":0.224,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":0.60299999999999998,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.309}]}]},{"osd":1,"up_from":8,"seq":34359740680,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:09:10 2023","interfaces":[{"interface":"back","average":{"1min":0.20100000000000001,"5min":0.19800000000000001,"15min":0.20000000000000001},"min":{"1min":0.16200000000000001,"5min":0.126,"15min":0.126},"max":{"1min":0.23899999999999999,"5min":0.437,"15min":0.437},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.22800000000000001,"5min":0.22900000000000001,"15min":0.22800000000000001},"min":{"1min":0.193,"5min":0.153,"15min":0.153},"max":{"1min":0.29199999999999998,"5min":0.439,"15min":0.45800000000000002},"last":0.23200000000000001}]},{"osd":2,"last update":"Sun Dec 24 21:09:10 2023","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":0.23599999999999999,"15min":0.24099999999999999},"min":{"1min":0.184,"5min":0.157,"15min":0.157},"max":{"1min":0.28399999999999997,"5min":0.45600000000000002,"15min":0.46999999999999997},"last":0.255},{"interface":"front","average":{"1min":0.27500000000000002,"5min":0.27100000000000002,"15min":0.26600000000000001},"min":{"1min":0.23400000000000001,"5min":0.192,"15min":0.16900000000000001},"max":{"1min":0.33500000000000002,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.313}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:09:39.145 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:39.145 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:39.145 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:39.145 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:39.146 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:39.146 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:39.146 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:39.146 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:39.146 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:39.147 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:39.147 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:39.147 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:39.147 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:39.147 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:09:39.447 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:39.451+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:39.470 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:39.474+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:39.852 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:39.856+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:40.490 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:40.493+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:40.490 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:40.494+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:40.878 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:40.882+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:41.449 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:41.453+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:41.466 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:41.470+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:41.864 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:41.867+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:42.436 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:42.440+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:42.496 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:42.500+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:42.824 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:42.828+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:43.474 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:43.477+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:43.525 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:43.529+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:43.838 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:43.842+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:44.505 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:44.509+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:44.511 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:44.515+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:44.813 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:44.817+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:45.462 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:45.465+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:45.554 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:45.558+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:45.833 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:45.837+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:46.460 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:46.464+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:46.573 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:46.577+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:46.813 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:46.817+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:47.495 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:47.499+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:47.617 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:47.621+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:47.776 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:47.780+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:48.515 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:48.519+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:48.571 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:48.575+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:48.786 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:48.791+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:49.492 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:49.497+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:49.525 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:49.528+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:49.745 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:49.749+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:50.484 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:50.488+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:50.503 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:50.508+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:50.780 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:50.783+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:51.443 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:51.446+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:51.519 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:51.524+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:51.811 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:51.816+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:52.401 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:52.405+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:52.560 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:52.565+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:52.840 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:52.845+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:53.424 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:53.429+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:53.573 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:53.578+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:53.805 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:53.809+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:54.390 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:54.395+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:54.605 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:54.610+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:54.772 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:54.777+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:55.380 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:55.385+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:55.590 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:55.594+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:55.724 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:55.729+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:56.366 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:56.371+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:56.597 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:56.602+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:56.733 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:56.737+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:57.353 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:57.358+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:57.551 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:57.556+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:57.697 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:57.702+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:58.374 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:58.379+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:58.598 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:58.602+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:58.674 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:58.679+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:59.148 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:09:59.416 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:09:59.421+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:59.497 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:09:59.499 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:09:59.511 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":5928,"stamp":"2023-12-24T21:09:58.661291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,4,6,10,518],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001927"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2826,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:26.705740+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T21:06:26.705740+0000","last_peered":"2023-12-24T21:06:26.705740+0000","last_clean":"2023-12-24T21:06:26.705740+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T21:06:26.705740+0000","last_undegraded":"2023-12-24T21:06:26.705740+0000","last_fullsized":"2023-12-24T21:06:26.705740+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3620,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:27.063375+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T21:06:27.063375+0000","last_peered":"2023-12-24T21:06:27.063375+0000","last_clean":"2023-12-24T21:06:27.063375+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T21:06:27.063375+0000","last_undegraded":"2023-12-24T21:06:27.063375+0000","last_fullsized":"2023-12-24T21:06:27.063375+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2983,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:27.404638+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T21:06:27.404638+0000","last_peered":"2023-12-24T21:06:27.404638+0000","last_clean":"2023-12-24T21:06:27.404638+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T21:06:27.404638+0000","last_undegraded":"2023-12-24T21:06:27.404638+0000","last_fullsized":"2023-12-24T21:06:27.404638+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10137,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:09:55.385636+0000","last_change":"2023-12-24T21:09:55.385636+0000","last_active":"2023-12-24T21:09:55.385636+0000","last_peered":"2023-12-24T21:09:55.385636+0000","last_clean":"2023-12-24T21:09:55.385636+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:09:55.385636+0000","last_undegraded":"2023-12-24T21:09:55.385636+0000","last_fullsized":"2023-12-24T21:09:55.385636+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3925,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:28.157180+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:06:28.157180+0000","last_peered":"2023-12-24T21:06:28.157180+0000","last_clean":"2023-12-24T21:06:28.157180+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:06:28.157180+0000","last_undegraded":"2023-12-24T21:06:28.157180+0000","last_fullsized":"2023-12-24T21:06:28.157180+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11205,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:28.497188+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T21:06:28.497188+0000","last_peered":"2023-12-24T21:06:28.497188+0000","last_clean":"2023-12-24T21:06:28.497188+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T21:06:28.497188+0000","last_undegraded":"2023-12-24T21:06:28.497188+0000","last_fullsized":"2023-12-24T21:06:28.497188+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":286,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:28.874717+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T21:06:28.874717+0000","last_peered":"2023-12-24T21:06:28.874717+0000","last_clean":"2023-12-24T21:06:28.874717+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T21:06:28.874717+0000","last_undegraded":"2023-12-24T21:06:28.874717+0000","last_fullsized":"2023-12-24T21:06:28.874717+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3531,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:29.208984+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T21:06:29.208984+0000","last_peered":"2023-12-24T21:06:29.208984+0000","last_clean":"2023-12-24T21:06:29.208984+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T21:06:29.208984+0000","last_undegraded":"2023-12-24T21:06:29.208984+0000","last_fullsized":"2023-12-24T21:06:29.208984+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10733,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:29.598630+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T21:06:29.598630+0000","last_peered":"2023-12-24T21:06:29.598630+0000","last_clean":"2023-12-24T21:06:29.598630+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T21:06:29.598630+0000","last_undegraded":"2023-12-24T21:06:29.598630+0000","last_fullsized":"2023-12-24T21:06:29.598630+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3509,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:06:29.987704+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:06:29.987704+0000","last_peered":"2023-12-24T21:06:29.987704+0000","last_clean":"2023-12-24T21:06:29.987704+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:06:29.987704+0000","last_undegraded":"2023-12-24T21:06:29.987704+0000","last_fullsized":"2023-12-24T21:06:29.987704+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 230s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12842,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:09:47.622916+0000","last_change":"2023-12-24T21:09:47.622916+0000","last_active":"2023-12-24T21:09:47.622916+0000","last_peered":"2023-12-24T21:09:47.622916+0000","last_clean":"2023-12-24T21:09:47.622916+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:09:47.622916+0000","last_undegraded":"2023-12-24T21:09:47.622916+0000","last_fullsized":"2023-12-24T21:09:47.622916+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3118,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:09:48.575819+0000","last_change":"2023-12-24T21:09:48.575819+0000","last_active":"2023-12-24T21:09:48.575819+0000","last_peered":"2023-12-24T21:09:48.575819+0000","last_clean":"2023-12-24T21:09:48.575819+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:09:48.575819+0000","last_undegraded":"2023-12-24T21:09:48.575819+0000","last_fullsized":"2023-12-24T21:09:48.575819+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3315,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:31.122557+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T21:06:31.122557+0000","last_peered":"2023-12-24T21:06:31.122557+0000","last_clean":"2023-12-24T21:06:31.122557+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T21:06:31.122557+0000","last_undegraded":"2023-12-24T21:06:31.122557+0000","last_fullsized":"2023-12-24T21:06:31.122557+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740682,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,4,6,10,6],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:09:02 2023","interfaces":[{"interface":"back","average":{"1min":0.23699999999999999,"5min":0.22800000000000001,"15min":0.219},"min":{"1min":0.159,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":1.0229999999999999,"5min":1.198,"15min":1.198},"last":0.192},{"interface":"front","average":{"1min":0.26400000000000001,"5min":0.248,"15min":0.24199999999999999},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16500000000000001},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.22700000000000001}]},{"osd":1,"last update":"Sun Dec 24 21:09:02 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.254,"15min":0.23999999999999999},"min":{"1min":0.187,"5min":0.187,"15min":0.17299999999999999},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.214},{"interface":"front","average":{"1min":0.29299999999999998,"5min":0.26900000000000002,"15min":0.26300000000000001},"min":{"1min":0.19900000000000001,"5min":0.191,"15min":0.17799999999999999},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.26400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359740681,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:09:08 2023","interfaces":[{"interface":"back","average":{"1min":0.186,"5min":0.189,"15min":0.19700000000000001},"min":{"1min":0.154,"5min":0.14999999999999999,"15min":0.13},"max":{"1min":0.22800000000000001,"5min":0.41099999999999998,"15min":0.42699999999999999},"last":0.154},{"interface":"front","average":{"1min":0.253,"5min":0.246,"15min":0.23300000000000001},"min":{"1min":0.189,"5min":0.14799999999999999,"15min":0.14499999999999999},"max":{"1min":0.55900000000000005,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.60499999999999998}]},{"osd":2,"last update":"Sun Dec 24 21:09:08 2023","interfaces":[{"interface":"back","average":{"1min":0.26800000000000002,"5min":0.26300000000000001,"15min":0.254},"min":{"1min":0.19800000000000001,"5min":0.17100000000000001,"15min":0.16800000000000001},"max":{"1min":0.58699999999999997,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.29799999999999999,"5min":0.29399999999999998,"15min":0.28199999999999997},"min":{"1min":0.224,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":0.60299999999999998,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.64700000000000002}]}]},{"osd":1,"up_from":8,"seq":34359740684,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:09:10 2023","interfaces":[{"interface":"back","average":{"1min":0.20100000000000001,"5min":0.19800000000000001,"15min":0.20000000000000001},"min":{"1min":0.16200000000000001,"5min":0.126,"15min":0.126},"max":{"1min":0.23899999999999999,"5min":0.437,"15min":0.437},"last":0.223},{"interface":"front","average":{"1min":0.22800000000000001,"5min":0.22900000000000001,"15min":0.22800000000000001},"min":{"1min":0.193,"5min":0.153,"15min":0.153},"max":{"1min":0.29199999999999998,"5min":0.439,"15min":0.45800000000000002},"last":0.26000000000000001}]},{"osd":2,"last update":"Sun Dec 24 21:09:10 2023","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":0.23599999999999999,"15min":0.24099999999999999},"min":{"1min":0.184,"5min":0.157,"15min":0.157},"max":{"1min":0.28399999999999997,"5min":0.45600000000000002,"15min":0.46999999999999997},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.27500000000000002,"5min":0.27100000000000002,"15min":0.26600000000000001},"min":{"1min":0.23400000000000001,"5min":0.192,"15min":0.16900000000000001},"max":{"1min":0.33500000000000002,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.308}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:09:59.512 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:59.512 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:59.513 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:59.513 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:59.513 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:59.513 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:59.514 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:59.514 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:59.514 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:59.514 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:59.514 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:59.515 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:59.515 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:09:59.515 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:09:59.584 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:09:59.588+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:09:59.658 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:09:59.663+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:00.394 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:00.399+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:00.550 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:00.555+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:00.706 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:00.711+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:01.403 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:01.407+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:01.523 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:01.528+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:01.744 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:01.749+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:02.434 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:02.439+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:02.493 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:02.498+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:02.709 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:02.714+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:03.450 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:03.455+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:03.456 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:03.461+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:03.752 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:03.757+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:04.459 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:04.463+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:04.479 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:04.483+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:04.796 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:04.799+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:05.485 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:05.488+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:05.489 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:05.493+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:05.750 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:05.754+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:06.444 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:06.448+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:06.492 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:06.496+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:06.727 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:06.731+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:07.476 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:07.480+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:07.520 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:07.524+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:07.700 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:07.704+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:08.447 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:08.450+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:08.550 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:08.554+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:08.668 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:08.671+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:09.418 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:09.421+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:09.598 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:09.602+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:09.687 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:09.690+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:10.454 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:10.458+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:10.620 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:10.623+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:10.658 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:10.661+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:11.468 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:11.471+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:11.615 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:11.619+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:11.633 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:11.636+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:12.471 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:12.475+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:12.578 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:12.582+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:12.670 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:12.674+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:13.501 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:13.504+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:13.612 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:13.615+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:13.698 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:13.701+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:14.514 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:14.518+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:14.640 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:14.644+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:14.734 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:14.738+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:15.516 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:15.520+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:15.662 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:15.666+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:15.766 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:15.770+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:16.562 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:16.566+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:16.671 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:16.674+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:16.778 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:16.782+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:17.595 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:17.598+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:17.695 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:17.699+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:17.814 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:17.818+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:18.620 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:18.623+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:18.729 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:18.733+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:18.835 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:18.839+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:19.516 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:10:19.631 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:19.634+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:19.729 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:19.733+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:19.811 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:19.815+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:19.869 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:10:19.871 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:10:19.882 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":5938,"stamp":"2023-12-24T21:10:18.664442+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,4,6,10,518],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001936"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2826,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:26.705740+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T21:06:26.705740+0000","last_peered":"2023-12-24T21:06:26.705740+0000","last_clean":"2023-12-24T21:06:26.705740+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T21:06:26.705740+0000","last_undegraded":"2023-12-24T21:06:26.705740+0000","last_fullsized":"2023-12-24T21:06:26.705740+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3620,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:27.063375+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T21:06:27.063375+0000","last_peered":"2023-12-24T21:06:27.063375+0000","last_clean":"2023-12-24T21:06:27.063375+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T21:06:27.063375+0000","last_undegraded":"2023-12-24T21:06:27.063375+0000","last_fullsized":"2023-12-24T21:06:27.063375+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2983,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:27.404638+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T21:06:27.404638+0000","last_peered":"2023-12-24T21:06:27.404638+0000","last_clean":"2023-12-24T21:06:27.404638+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T21:06:27.404638+0000","last_undegraded":"2023-12-24T21:06:27.404638+0000","last_fullsized":"2023-12-24T21:06:27.404638+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10143,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:10.624688+0000","last_change":"2023-12-24T21:10:10.624688+0000","last_active":"2023-12-24T21:10:10.624688+0000","last_peered":"2023-12-24T21:10:10.624688+0000","last_clean":"2023-12-24T21:10:10.624688+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:10:10.624688+0000","last_undegraded":"2023-12-24T21:10:10.624688+0000","last_fullsized":"2023-12-24T21:10:10.624688+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3925,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:28.157180+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:06:28.157180+0000","last_peered":"2023-12-24T21:06:28.157180+0000","last_clean":"2023-12-24T21:06:28.157180+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:06:28.157180+0000","last_undegraded":"2023-12-24T21:06:28.157180+0000","last_fullsized":"2023-12-24T21:06:28.157180+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11205,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:28.497188+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T21:06:28.497188+0000","last_peered":"2023-12-24T21:06:28.497188+0000","last_clean":"2023-12-24T21:06:28.497188+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T21:06:28.497188+0000","last_undegraded":"2023-12-24T21:06:28.497188+0000","last_fullsized":"2023-12-24T21:06:28.497188+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":286,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:28.874717+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T21:06:28.874717+0000","last_peered":"2023-12-24T21:06:28.874717+0000","last_clean":"2023-12-24T21:06:28.874717+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T21:06:28.874717+0000","last_undegraded":"2023-12-24T21:06:28.874717+0000","last_fullsized":"2023-12-24T21:06:28.874717+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3531,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:29.208984+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T21:06:29.208984+0000","last_peered":"2023-12-24T21:06:29.208984+0000","last_clean":"2023-12-24T21:06:29.208984+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T21:06:29.208984+0000","last_undegraded":"2023-12-24T21:06:29.208984+0000","last_fullsized":"2023-12-24T21:06:29.208984+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10733,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:29.598630+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T21:06:29.598630+0000","last_peered":"2023-12-24T21:06:29.598630+0000","last_clean":"2023-12-24T21:06:29.598630+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T21:06:29.598630+0000","last_undegraded":"2023-12-24T21:06:29.598630+0000","last_fullsized":"2023-12-24T21:06:29.598630+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3509,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:06:29.987704+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:06:29.987704+0000","last_peered":"2023-12-24T21:06:29.987704+0000","last_clean":"2023-12-24T21:06:29.987704+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:06:29.987704+0000","last_undegraded":"2023-12-24T21:06:29.987704+0000","last_fullsized":"2023-12-24T21:06:29.987704+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 230s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12851,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:12.674993+0000","last_change":"2023-12-24T21:10:12.674993+0000","last_active":"2023-12-24T21:10:12.674993+0000","last_peered":"2023-12-24T21:10:12.674993+0000","last_clean":"2023-12-24T21:10:12.674993+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:10:12.674993+0000","last_undegraded":"2023-12-24T21:10:12.674993+0000","last_fullsized":"2023-12-24T21:10:12.674993+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3127,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:16.783667+0000","last_change":"2023-12-24T21:10:16.783667+0000","last_active":"2023-12-24T21:10:16.783667+0000","last_peered":"2023-12-24T21:10:16.783667+0000","last_clean":"2023-12-24T21:10:16.783667+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:10:16.783667+0000","last_undegraded":"2023-12-24T21:10:16.783667+0000","last_fullsized":"2023-12-24T21:10:16.783667+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3315,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:31.122557+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T21:06:31.122557+0000","last_peered":"2023-12-24T21:06:31.122557+0000","last_clean":"2023-12-24T21:06:31.122557+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T21:06:31.122557+0000","last_undegraded":"2023-12-24T21:06:31.122557+0000","last_fullsized":"2023-12-24T21:06:31.122557+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740686,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,4,6,10,6],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:10:05 2023","interfaces":[{"interface":"back","average":{"1min":0.21099999999999999,"5min":0.22700000000000001,"15min":0.22},"min":{"1min":0.16400000000000001,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":0.39900000000000002,"5min":1.198,"15min":1.198},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.23300000000000001,"5min":0.24299999999999999,"15min":0.24199999999999999},"min":{"1min":0.182,"5min":0.16900000000000001,"15min":0.16500000000000001},"max":{"1min":0.435,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.33900000000000002}]},{"osd":1,"last update":"Sun Dec 24 21:10:05 2023","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":0.252,"15min":0.24199999999999999},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17299999999999999},"max":{"1min":0.497,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.26500000000000001,"5min":0.26800000000000002,"15min":0.26400000000000001},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.47099999999999997,"5min":1.153,"15min":1.153},"last":0.372}]}]},{"osd":0,"up_from":8,"seq":34359740685,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:10:10 2023","interfaces":[{"interface":"back","average":{"1min":0.19,"5min":0.189,"15min":0.19700000000000001},"min":{"1min":0.14499999999999999,"5min":0.14499999999999999,"15min":0.13},"max":{"1min":0.39700000000000002,"5min":0.41099999999999998,"15min":0.42699999999999999},"last":0.219},{"interface":"front","average":{"1min":0.253,"5min":0.252,"15min":0.23499999999999999},"min":{"1min":0.16200000000000001,"5min":0.14799999999999999,"15min":0.14499999999999999},"max":{"1min":0.60499999999999998,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.66300000000000003}]},{"osd":2,"last update":"Sun Dec 24 21:10:10 2023","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.26800000000000002,"15min":0.255},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.16800000000000001},"max":{"1min":0.63100000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.30099999999999999,"15min":0.28399999999999997},"min":{"1min":0.192,"5min":0.192,"15min":0.188},"max":{"1min":0.64700000000000002,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.70899999999999996}]}]},{"osd":1,"up_from":8,"seq":34359740688,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:10:14 2023","interfaces":[{"interface":"back","average":{"1min":0.18099999999999999,"5min":0.192,"15min":0.20000000000000001},"min":{"1min":0.14199999999999999,"5min":0.126,"15min":0.126},"max":{"1min":0.26500000000000001,"5min":0.437,"15min":0.437},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.214,"5min":0.222,"15min":0.22700000000000001},"min":{"1min":0.17699999999999999,"5min":0.153,"15min":0.153},"max":{"1min":0.26000000000000001,"5min":0.41299999999999998,"15min":0.45800000000000002},"last":0.16900000000000001}]},{"osd":2,"last update":"Sun Dec 24 21:10:14 2023","interfaces":[{"interface":"back","average":{"1min":0.20899999999999999,"5min":0.224,"15min":0.23899999999999999},"min":{"1min":0.16500000000000001,"5min":0.157,"15min":0.157},"max":{"1min":0.29699999999999999,"5min":0.42699999999999999,"15min":0.46999999999999997},"last":0.183},{"interface":"front","average":{"1min":0.25600000000000001,"5min":0.26500000000000001,"15min":0.26600000000000001},"min":{"1min":0.216,"5min":0.192,"15min":0.16900000000000001},"max":{"1min":0.313,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.23699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:10:19.883 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:10:19.883 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:10:19.884 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:10:19.884 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:10:19.884 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:10:19.884 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:10:19.885 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:10:19.885 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:10:19.885 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:10:19.885 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:10:19.885 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:10:19.886 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:10:19.886 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:10:19.886 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:10:20.587 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:20.590+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:20.713 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:20.717+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:20.860 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:20.864+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:21.569 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:21.572+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:21.680 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:21.684+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:21.900 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:21.903+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:22.601 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:22.604+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:22.723 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:22.726+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:22.937 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:22.941+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:23.601 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:23.604+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:23.772 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:23.776+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:23.904 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:23.907+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:24.637 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:24.640+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:24.810 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:24.813+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:24.880 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:24.884+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:25.656 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:25.660+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:25.858 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:25.862+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:25.886 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:25.890+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:26.668 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:26.671+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:26.844 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:26.848+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:26.870 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:26.873+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:27.704 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:27.707+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:27.826 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:27.830+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:27.836 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:27.840+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:28.743 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:28.747+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:28.808 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:28.812+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:28.810 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:28.814+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:29.787 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:29.790+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:29.804 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:29.808+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:29.824 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:29.828+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:30.742 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:30.745+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:30.792 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:30.796+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:30.801 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:30.805+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:31.768 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:31.772+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:31.834 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:31.838+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:31.844 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:31.848+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:32.803 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:32.806+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:32.812 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:32.816+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:32.832 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:32.835+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:33.793 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:33.796+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:33.838 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:33.842+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:33.849 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:33.853+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:34.798 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:34.801+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:34.820 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:34.824+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:34.853 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:34.857+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:35.839 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:35.842+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:35.862 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:35.865+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:35.863 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:35.867+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:36.826 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:36.830+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:36.841 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:36.846+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:36.901 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:36.905+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:37.794 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:37.799+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:37.801 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:37.806+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:37.925 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:37.929+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:38.778 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:38.782+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:38.834 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:38.838+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:38.883 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:38.887+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:39.732 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:39.736+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:39.856 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:39.860+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:39.867 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:39.872+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:39.887 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:10:40.248 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:10:40.251 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:10:40.263 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":5948,"stamp":"2023-12-24T21:10:38.667614+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,4,6,10,518],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001987"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2826,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:26.705740+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T21:06:26.705740+0000","last_peered":"2023-12-24T21:06:26.705740+0000","last_clean":"2023-12-24T21:06:26.705740+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T21:06:26.705740+0000","last_undegraded":"2023-12-24T21:06:26.705740+0000","last_fullsized":"2023-12-24T21:06:26.705740+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3620,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:27.063375+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T21:06:27.063375+0000","last_peered":"2023-12-24T21:06:27.063375+0000","last_clean":"2023-12-24T21:06:27.063375+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T21:06:27.063375+0000","last_undegraded":"2023-12-24T21:06:27.063375+0000","last_fullsized":"2023-12-24T21:06:27.063375+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2983,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:27.404638+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T21:06:27.404638+0000","last_peered":"2023-12-24T21:06:27.404638+0000","last_clean":"2023-12-24T21:06:27.404638+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T21:06:27.404638+0000","last_undegraded":"2023-12-24T21:06:27.404638+0000","last_fullsized":"2023-12-24T21:06:27.404638+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10149,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:29.829014+0000","last_change":"2023-12-24T21:10:29.829014+0000","last_active":"2023-12-24T21:10:29.829014+0000","last_peered":"2023-12-24T21:10:29.829014+0000","last_clean":"2023-12-24T21:10:29.829014+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:10:29.829014+0000","last_undegraded":"2023-12-24T21:10:29.829014+0000","last_fullsized":"2023-12-24T21:10:29.829014+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3925,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:28.157180+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:06:28.157180+0000","last_peered":"2023-12-24T21:06:28.157180+0000","last_clean":"2023-12-24T21:06:28.157180+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:06:28.157180+0000","last_undegraded":"2023-12-24T21:06:28.157180+0000","last_fullsized":"2023-12-24T21:06:28.157180+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11205,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:28.497188+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T21:06:28.497188+0000","last_peered":"2023-12-24T21:06:28.497188+0000","last_clean":"2023-12-24T21:06:28.497188+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T21:06:28.497188+0000","last_undegraded":"2023-12-24T21:06:28.497188+0000","last_fullsized":"2023-12-24T21:06:28.497188+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":286,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:28.874717+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T21:06:28.874717+0000","last_peered":"2023-12-24T21:06:28.874717+0000","last_clean":"2023-12-24T21:06:28.874717+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T21:06:28.874717+0000","last_undegraded":"2023-12-24T21:06:28.874717+0000","last_fullsized":"2023-12-24T21:06:28.874717+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3531,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:29.208984+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T21:06:29.208984+0000","last_peered":"2023-12-24T21:06:29.208984+0000","last_clean":"2023-12-24T21:06:29.208984+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T21:06:29.208984+0000","last_undegraded":"2023-12-24T21:06:29.208984+0000","last_fullsized":"2023-12-24T21:06:29.208984+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10733,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:29.598630+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T21:06:29.598630+0000","last_peered":"2023-12-24T21:06:29.598630+0000","last_clean":"2023-12-24T21:06:29.598630+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T21:06:29.598630+0000","last_undegraded":"2023-12-24T21:06:29.598630+0000","last_fullsized":"2023-12-24T21:06:29.598630+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3509,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:06:29.987704+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:06:29.987704+0000","last_peered":"2023-12-24T21:06:29.987704+0000","last_clean":"2023-12-24T21:06:29.987704+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:06:29.987704+0000","last_undegraded":"2023-12-24T21:06:29.987704+0000","last_fullsized":"2023-12-24T21:06:29.987704+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 230s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12857,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:35.843207+0000","last_change":"2023-12-24T21:10:35.843207+0000","last_active":"2023-12-24T21:10:35.843207+0000","last_peered":"2023-12-24T21:10:35.843207+0000","last_clean":"2023-12-24T21:10:35.843207+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:10:35.843207+0000","last_undegraded":"2023-12-24T21:10:35.843207+0000","last_fullsized":"2023-12-24T21:10:35.843207+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3130,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:22.941653+0000","last_change":"2023-12-24T21:10:22.941653+0000","last_active":"2023-12-24T21:10:22.941653+0000","last_peered":"2023-12-24T21:10:22.941653+0000","last_clean":"2023-12-24T21:10:22.941653+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:10:22.941653+0000","last_undegraded":"2023-12-24T21:10:22.941653+0000","last_fullsized":"2023-12-24T21:10:22.941653+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3315,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:06:31.122557+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T21:06:31.122557+0000","last_peered":"2023-12-24T21:06:31.122557+0000","last_clean":"2023-12-24T21:06:31.122557+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T21:06:31.122557+0000","last_undegraded":"2023-12-24T21:06:31.122557+0000","last_fullsized":"2023-12-24T21:06:31.122557+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740690,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,4,6,10,6],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:10:05 2023","interfaces":[{"interface":"back","average":{"1min":0.21099999999999999,"5min":0.22700000000000001,"15min":0.22},"min":{"1min":0.16400000000000001,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":0.39900000000000002,"5min":1.198,"15min":1.198},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.23300000000000001,"5min":0.24299999999999999,"15min":0.24199999999999999},"min":{"1min":0.182,"5min":0.16900000000000001,"15min":0.16500000000000001},"max":{"1min":0.435,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.26500000000000001}]},{"osd":1,"last update":"Sun Dec 24 21:10:05 2023","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":0.252,"15min":0.24199999999999999},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17299999999999999},"max":{"1min":0.497,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.26500000000000001,"5min":0.26800000000000002,"15min":0.26400000000000001},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.47099999999999997,"5min":1.153,"15min":1.153},"last":0.309}]}]},{"osd":0,"up_from":8,"seq":34359740689,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:10:10 2023","interfaces":[{"interface":"back","average":{"1min":0.19,"5min":0.189,"15min":0.19700000000000001},"min":{"1min":0.14499999999999999,"5min":0.14499999999999999,"15min":0.13},"max":{"1min":0.39700000000000002,"5min":0.41099999999999998,"15min":0.42699999999999999},"last":0.17399999999999999},{"interface":"front","average":{"1min":0.253,"5min":0.252,"15min":0.23499999999999999},"min":{"1min":0.16200000000000001,"5min":0.14799999999999999,"15min":0.14499999999999999},"max":{"1min":0.60499999999999998,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.21099999999999999}]},{"osd":2,"last update":"Sun Dec 24 21:10:10 2023","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.26800000000000002,"15min":0.255},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.16800000000000001},"max":{"1min":0.63100000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.184},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.30099999999999999,"15min":0.28399999999999997},"min":{"1min":0.192,"5min":0.192,"15min":0.188},"max":{"1min":0.64700000000000002,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.23200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359740692,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:10:14 2023","interfaces":[{"interface":"back","average":{"1min":0.18099999999999999,"5min":0.192,"15min":0.20000000000000001},"min":{"1min":0.14199999999999999,"5min":0.126,"15min":0.126},"max":{"1min":0.26500000000000001,"5min":0.437,"15min":0.437},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.214,"5min":0.222,"15min":0.22700000000000001},"min":{"1min":0.17699999999999999,"5min":0.153,"15min":0.153},"max":{"1min":0.26000000000000001,"5min":0.41299999999999998,"15min":0.45800000000000002},"last":0.19900000000000001}]},{"osd":2,"last update":"Sun Dec 24 21:10:14 2023","interfaces":[{"interface":"back","average":{"1min":0.20899999999999999,"5min":0.224,"15min":0.23899999999999999},"min":{"1min":0.16500000000000001,"5min":0.157,"15min":0.157},"max":{"1min":0.29699999999999999,"5min":0.42699999999999999,"15min":0.46999999999999997},"last":0.17999999999999999},{"interface":"front","average":{"1min":0.25600000000000001,"5min":0.26500000000000001,"15min":0.26600000000000001},"min":{"1min":0.216,"5min":0.192,"15min":0.16900000000000001},"max":{"1min":0.313,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.23499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:10:40.264 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:10:40.264 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:10:40.265 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:10:40.265 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:10:40.265 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:10:40.265 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:10:40.265 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:10:40.265 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:10:40.266 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:10:40.266 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:10:40.266 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:10:40.266 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:10:40.266 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:10:40.266 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.7 2023-12-24T21:10:40.635 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.7 on osd.1 to deep-scrub 2023-12-24T21:10:40.647 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.6 2023-12-24T21:10:40.700 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:40.704+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:40.857 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:40.862+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:40.872 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:40.876+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:41.018 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.6 on osd.1 to deep-scrub 2023-12-24T21:10:41.030 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2023-12-24T21:10:41.366 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2023-12-24T21:10:41.378 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-12-24T21:10:41.692 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:41.696+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:41.741 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-12-24T21:10:41.752 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.0 2023-12-24T21:10:41.868 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:41.872+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:41.904 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:41.908+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:42.098 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.0 on osd.2 to deep-scrub 2023-12-24T21:10:42.109 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-12-24T21:10:42.470 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-12-24T21:10:42.482 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.0 2023-12-24T21:10:42.688 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:42.692+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:42.851 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 1.0 on osd.1 to deep-scrub 2023-12-24T21:10:42.860 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:42.865+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:42.861 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:42.865+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:42.863 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.3 2023-12-24T21:10:43.224 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.3 on osd.1 to deep-scrub 2023-12-24T21:10:43.236 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-12-24T21:10:43.594 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-12-24T21:10:43.606 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.1 2023-12-24T21:10:43.701 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:43.706+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:43.823 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:43.828+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:43.899 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:43.903+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:43.990 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.1 on osd.2 to deep-scrub 2023-12-24T21:10:44.002 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-12-24T21:10:44.378 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-12-24T21:10:44.390 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.2 2023-12-24T21:10:44.670 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:44.674+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:44.764 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.2 on osd.0 to deep-scrub 2023-12-24T21:10:44.776 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.4 2023-12-24T21:10:44.808 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:44.812+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:44.913 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:44.917+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:45.106 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.4 on osd.1 to deep-scrub 2023-12-24T21:10:45.118 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:10:45.679 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:45.683+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:45.826 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:45.830+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:45.911 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:45.915+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:46.669 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:46.672+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:46.821 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:46.825+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:46.885 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:46.889+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:47.644 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:47.647+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:47.823 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:47.827+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:47.892 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:47.896+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:48.667 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:48.671+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:48.862 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:48.866+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:48.918 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:48.922+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:49.695 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:49.698+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:49.879 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:49.882+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:49.883 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:49.886+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:50.709 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:50.712+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:50.852 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:50.855+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:50.906 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:50.909+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:51.742 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:51.744+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:51.854 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:51.857+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:51.863 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:51.866+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:52.781 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:52.784+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:52.868 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:52.871+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:52.869 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:52.873+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:53.760 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:53.763+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:53.844 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:53.847+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:53.882 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:53.885+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:54.798 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:54.801+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:54.818 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:54.821+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:54.841 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:54.844+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:55.780 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:55.783+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:55.811 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:55.814+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:55.841 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:55.844+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:56.764 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:56.767+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:56.766 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:56.769+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:56.794 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:56.797+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:57.786 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:57.787+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:57.790 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:57.793+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:57.806 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:57.809+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:58.739 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:58.742+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:58.817 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:58.820+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:58.852 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:58.855+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:59.692 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:10:59.695+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:59.804 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:10:59.807+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:10:59.864 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:10:59.867+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:00.649 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:00.652+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:00.831 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:00.834+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:00.856 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:00.859+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:01.690 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:01.693+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:01.820 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:01.823+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:01.903 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:01.906+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:02.680 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:02.682+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:02.820 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:02.823+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:02.878 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:02.881+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:03.648 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:03.651+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:03.850 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:03.853+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:03.928 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:03.931+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:04.608 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:04.611+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:04.899 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:04.902+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:04.963 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:04.966+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:05.119 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:11:05.479 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:11:05.482 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:11:05.493 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":5961,"stamp":"2023-12-24T21:11:04.671438+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,8,10,518],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001729"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2827,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:33.675344+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T21:08:33.675344+0000","last_peered":"2023-12-24T21:08:33.675344+0000","last_clean":"2023-12-24T21:08:33.675344+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T21:08:33.675344+0000","last_undegraded":"2023-12-24T21:08:33.675344+0000","last_fullsized":"2023-12-24T21:08:33.675344+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3621,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:34.024281+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T21:08:34.024281+0000","last_peered":"2023-12-24T21:08:34.024281+0000","last_clean":"2023-12-24T21:08:34.024281+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T21:08:34.024281+0000","last_undegraded":"2023-12-24T21:08:34.024281+0000","last_fullsized":"2023-12-24T21:08:34.024281+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2984,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:34.391689+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T21:08:34.391689+0000","last_peered":"2023-12-24T21:08:34.391689+0000","last_clean":"2023-12-24T21:08:34.391689+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T21:08:34.391689+0000","last_undegraded":"2023-12-24T21:08:34.391689+0000","last_fullsized":"2023-12-24T21:08:34.391689+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10156,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:57.794310+0000","last_change":"2023-12-24T21:10:57.794310+0000","last_active":"2023-12-24T21:10:57.794310+0000","last_peered":"2023-12-24T21:10:57.794310+0000","last_clean":"2023-12-24T21:10:57.794310+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:10:57.794310+0000","last_undegraded":"2023-12-24T21:10:57.794310+0000","last_fullsized":"2023-12-24T21:10:57.794310+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3926,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:35.134823+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:08:35.134823+0000","last_peered":"2023-12-24T21:08:35.134823+0000","last_clean":"2023-12-24T21:08:35.134823+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:08:35.134823+0000","last_undegraded":"2023-12-24T21:08:35.134823+0000","last_fullsized":"2023-12-24T21:08:35.134823+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11206,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:35.505493+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T21:08:35.505493+0000","last_peered":"2023-12-24T21:08:35.505493+0000","last_clean":"2023-12-24T21:08:35.505493+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T21:08:35.505493+0000","last_undegraded":"2023-12-24T21:08:35.505493+0000","last_fullsized":"2023-12-24T21:08:35.505493+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":287,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:35.841800+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T21:08:35.841800+0000","last_peered":"2023-12-24T21:08:35.841800+0000","last_clean":"2023-12-24T21:08:35.841800+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T21:08:35.841800+0000","last_undegraded":"2023-12-24T21:08:35.841800+0000","last_fullsized":"2023-12-24T21:08:35.841800+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3532,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:36.195031+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T21:08:36.195031+0000","last_peered":"2023-12-24T21:08:36.195031+0000","last_clean":"2023-12-24T21:08:36.195031+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T21:08:36.195031+0000","last_undegraded":"2023-12-24T21:08:36.195031+0000","last_fullsized":"2023-12-24T21:08:36.195031+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10734,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:36.544069+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T21:08:36.544069+0000","last_peered":"2023-12-24T21:08:36.544069+0000","last_clean":"2023-12-24T21:08:36.544069+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T21:08:36.544069+0000","last_undegraded":"2023-12-24T21:08:36.544069+0000","last_fullsized":"2023-12-24T21:08:36.544069+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3510,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:08:36.929523+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:08:36.929523+0000","last_peered":"2023-12-24T21:08:36.929523+0000","last_clean":"2023-12-24T21:08:36.929523+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:08:36.929523+0000","last_undegraded":"2023-12-24T21:08:36.929523+0000","last_fullsized":"2023-12-24T21:08:36.929523+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 357s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12864,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:59.867824+0000","last_change":"2023-12-24T21:10:59.867824+0000","last_active":"2023-12-24T21:10:59.867824+0000","last_peered":"2023-12-24T21:10:59.867824+0000","last_clean":"2023-12-24T21:10:59.867824+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:10:59.867824+0000","last_undegraded":"2023-12-24T21:10:59.867824+0000","last_fullsized":"2023-12-24T21:10:59.867824+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3137,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:55.783922+0000","last_change":"2023-12-24T21:10:55.783922+0000","last_active":"2023-12-24T21:10:55.783922+0000","last_peered":"2023-12-24T21:10:55.783922+0000","last_clean":"2023-12-24T21:10:55.783922+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:10:55.783922+0000","last_undegraded":"2023-12-24T21:10:55.783922+0000","last_fullsized":"2023-12-24T21:10:55.783922+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3316,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:37.978133+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T21:08:37.978133+0000","last_peered":"2023-12-24T21:08:37.978133+0000","last_clean":"2023-12-24T21:08:37.978133+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T21:08:37.978133+0000","last_undegraded":"2023-12-24T21:08:37.978133+0000","last_fullsized":"2023-12-24T21:08:37.978133+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740695,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,8,10,6],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:10:05 2023","interfaces":[{"interface":"back","average":{"1min":0.21099999999999999,"5min":0.22700000000000001,"15min":0.22},"min":{"1min":0.16400000000000001,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":0.39900000000000002,"5min":1.198,"15min":1.198},"last":0.16800000000000001},{"interface":"front","average":{"1min":0.23300000000000001,"5min":0.24299999999999999,"15min":0.24199999999999999},"min":{"1min":0.182,"5min":0.16900000000000001,"15min":0.16500000000000001},"max":{"1min":0.435,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.19900000000000001}]},{"osd":1,"last update":"Sun Dec 24 21:10:05 2023","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":0.252,"15min":0.24199999999999999},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17299999999999999},"max":{"1min":0.497,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.26500000000000001,"5min":0.26800000000000002,"15min":0.26400000000000001},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.47099999999999997,"5min":1.153,"15min":1.153},"last":0.23799999999999999}]}]},{"osd":0,"up_from":8,"seq":34359740694,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:10:10 2023","interfaces":[{"interface":"back","average":{"1min":0.19,"5min":0.189,"15min":0.19700000000000001},"min":{"1min":0.14499999999999999,"5min":0.14499999999999999,"15min":0.13},"max":{"1min":0.39700000000000002,"5min":0.41099999999999998,"15min":0.42699999999999999},"last":0.189},{"interface":"front","average":{"1min":0.253,"5min":0.252,"15min":0.23499999999999999},"min":{"1min":0.16200000000000001,"5min":0.14799999999999999,"15min":0.14499999999999999},"max":{"1min":0.60499999999999998,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.20999999999999999}]},{"osd":2,"last update":"Sun Dec 24 21:10:10 2023","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.26800000000000002,"15min":0.255},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.16800000000000001},"max":{"1min":0.63100000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.30099999999999999,"15min":0.28399999999999997},"min":{"1min":0.192,"5min":0.192,"15min":0.188},"max":{"1min":0.64700000000000002,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.24099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359740697,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:10:14 2023","interfaces":[{"interface":"back","average":{"1min":0.18099999999999999,"5min":0.192,"15min":0.20000000000000001},"min":{"1min":0.14199999999999999,"5min":0.126,"15min":0.126},"max":{"1min":0.26500000000000001,"5min":0.437,"15min":0.437},"last":0.20699999999999999},{"interface":"front","average":{"1min":0.214,"5min":0.222,"15min":0.22700000000000001},"min":{"1min":0.17699999999999999,"5min":0.153,"15min":0.153},"max":{"1min":0.26000000000000001,"5min":0.41299999999999998,"15min":0.45800000000000002},"last":0.23899999999999999}]},{"osd":2,"last update":"Sun Dec 24 21:10:14 2023","interfaces":[{"interface":"back","average":{"1min":0.20899999999999999,"5min":0.224,"15min":0.23899999999999999},"min":{"1min":0.16500000000000001,"5min":0.157,"15min":0.157},"max":{"1min":0.29699999999999999,"5min":0.42699999999999999,"15min":0.46999999999999997},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.25600000000000001,"5min":0.26500000000000001,"15min":0.26600000000000001},"min":{"1min":0.216,"5min":0.192,"15min":0.16900000000000001},"max":{"1min":0.313,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.26800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:11:05.494 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:05.494 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:05.495 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:05.495 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:05.495 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:05.495 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:05.495 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:05.495 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:05.495 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:05.496 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:05.496 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:05.496 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:05.496 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:05.496 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:11:05.642 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:05.646+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:05.920 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:05.923+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:05.990 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:05.994+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:06.691 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:06.694+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:06.891 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:06.894+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:07.033 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:07.036+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:07.670 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:07.673+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:07.842 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:07.845+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:08.072 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:08.075+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:08.648 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:08.650+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:08.845 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:08.849+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:09.090 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:09.093+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:09.631 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:09.634+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:09.830 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:09.833+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:10.078 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:10.081+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:10.616 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:10.619+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:10.813 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:10.816+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:11.113 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:11.116+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:11.599 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:11.602+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:11.803 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:11.806+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:12.154 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:12.157+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:12.648 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:12.651+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:12.755 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:12.758+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:13.124 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:13.126+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:13.668 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:13.671+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:13.736 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:13.739+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:14.112 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:14.115+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:14.712 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:14.715+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:14.718 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:14.721+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:15.158 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:15.161+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:15.718 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:15.721+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:15.745 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:15.748+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:16.148 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:16.151+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:16.712 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:16.715+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:16.748 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:16.751+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:17.180 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:17.183+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:17.717 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:17.720+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:17.784 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:17.787+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:18.167 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:18.170+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:18.691 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:18.694+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:18.821 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:18.824+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:19.188 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:19.190+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:19.697 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:19.700+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:19.866 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:19.869+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:20.225 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:20.228+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:20.722 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:20.724+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:20.835 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:20.837+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:21.245 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:21.249+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:21.709 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:21.713+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:21.798 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:21.802+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:22.201 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:22.205+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:22.683 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:22.687+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:22.821 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:22.825+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:23.230 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:23.233+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:23.681 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:23.684+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:23.856 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:23.860+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:24.182 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:24.186+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:24.669 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:24.672+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:24.867 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:24.871+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:25.164 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:25.168+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:25.497 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:11:25.708 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:25.712+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 28 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:25.829 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:11:25.834 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:11:25.844 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":5971,"stamp":"2023-12-24T21:11:24.674698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,8,10,518],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002094"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2827,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:33.675344+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T21:08:33.675344+0000","last_peered":"2023-12-24T21:08:33.675344+0000","last_clean":"2023-12-24T21:08:33.675344+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T21:08:33.675344+0000","last_undegraded":"2023-12-24T21:08:33.675344+0000","last_fullsized":"2023-12-24T21:08:33.675344+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3621,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:34.024281+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T21:08:34.024281+0000","last_peered":"2023-12-24T21:08:34.024281+0000","last_clean":"2023-12-24T21:08:34.024281+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T21:08:34.024281+0000","last_undegraded":"2023-12-24T21:08:34.024281+0000","last_fullsized":"2023-12-24T21:08:34.024281+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2984,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:34.391689+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T21:08:34.391689+0000","last_peered":"2023-12-24T21:08:34.391689+0000","last_clean":"2023-12-24T21:08:34.391689+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T21:08:34.391689+0000","last_undegraded":"2023-12-24T21:08:34.391689+0000","last_fullsized":"2023-12-24T21:08:34.391689+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10165,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:11:19.191664+0000","last_change":"2023-12-24T21:11:19.191664+0000","last_active":"2023-12-24T21:11:19.191664+0000","last_peered":"2023-12-24T21:11:19.191664+0000","last_clean":"2023-12-24T21:11:19.191664+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:11:19.191664+0000","last_undegraded":"2023-12-24T21:11:19.191664+0000","last_fullsized":"2023-12-24T21:11:19.191664+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3926,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:35.134823+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:08:35.134823+0000","last_peered":"2023-12-24T21:08:35.134823+0000","last_clean":"2023-12-24T21:08:35.134823+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:08:35.134823+0000","last_undegraded":"2023-12-24T21:08:35.134823+0000","last_fullsized":"2023-12-24T21:08:35.134823+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11206,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:35.505493+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T21:08:35.505493+0000","last_peered":"2023-12-24T21:08:35.505493+0000","last_clean":"2023-12-24T21:08:35.505493+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T21:08:35.505493+0000","last_undegraded":"2023-12-24T21:08:35.505493+0000","last_fullsized":"2023-12-24T21:08:35.505493+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":287,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:35.841800+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T21:08:35.841800+0000","last_peered":"2023-12-24T21:08:35.841800+0000","last_clean":"2023-12-24T21:08:35.841800+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T21:08:35.841800+0000","last_undegraded":"2023-12-24T21:08:35.841800+0000","last_fullsized":"2023-12-24T21:08:35.841800+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3532,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:36.195031+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T21:08:36.195031+0000","last_peered":"2023-12-24T21:08:36.195031+0000","last_clean":"2023-12-24T21:08:36.195031+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T21:08:36.195031+0000","last_undegraded":"2023-12-24T21:08:36.195031+0000","last_fullsized":"2023-12-24T21:08:36.195031+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10734,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:36.544069+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T21:08:36.544069+0000","last_peered":"2023-12-24T21:08:36.544069+0000","last_clean":"2023-12-24T21:08:36.544069+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T21:08:36.544069+0000","last_undegraded":"2023-12-24T21:08:36.544069+0000","last_fullsized":"2023-12-24T21:08:36.544069+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3510,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:08:36.929523+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:08:36.929523+0000","last_peered":"2023-12-24T21:08:36.929523+0000","last_clean":"2023-12-24T21:08:36.929523+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:08:36.929523+0000","last_undegraded":"2023-12-24T21:08:36.929523+0000","last_fullsized":"2023-12-24T21:08:36.929523+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 357s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12873,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:11:21.249013+0000","last_change":"2023-12-24T21:11:21.249013+0000","last_active":"2023-12-24T21:11:21.249013+0000","last_peered":"2023-12-24T21:11:21.249013+0000","last_clean":"2023-12-24T21:11:21.249013+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:11:21.249013+0000","last_undegraded":"2023-12-24T21:11:21.249013+0000","last_fullsized":"2023-12-24T21:11:21.249013+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3146,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:11:17.184143+0000","last_change":"2023-12-24T21:11:17.184143+0000","last_active":"2023-12-24T21:11:17.184143+0000","last_peered":"2023-12-24T21:11:17.184143+0000","last_clean":"2023-12-24T21:11:17.184143+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:11:17.184143+0000","last_undegraded":"2023-12-24T21:11:17.184143+0000","last_fullsized":"2023-12-24T21:11:17.184143+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3316,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:37.978133+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T21:08:37.978133+0000","last_peered":"2023-12-24T21:08:37.978133+0000","last_clean":"2023-12-24T21:08:37.978133+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T21:08:37.978133+0000","last_undegraded":"2023-12-24T21:08:37.978133+0000","last_fullsized":"2023-12-24T21:08:37.978133+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740699,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,8,10,6],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:11:06 2023","interfaces":[{"interface":"back","average":{"1min":0.23499999999999999,"5min":0.23000000000000001,"15min":0.222},"min":{"1min":0.16800000000000001,"5min":0.14899999999999999,"15min":0.14399999999999999},"max":{"1min":0.44,"5min":1.198,"15min":1.198},"last":0.183},{"interface":"front","average":{"1min":0.249,"5min":0.245,"15min":0.24399999999999999},"min":{"1min":0.182,"5min":0.16900000000000001,"15min":0.16500000000000001},"max":{"1min":0.39400000000000002,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.20899999999999999}]},{"osd":1,"last update":"Sun Dec 24 21:11:06 2023","interfaces":[{"interface":"back","average":{"1min":0.254,"5min":0.252,"15min":0.24399999999999999},"min":{"1min":0.18099999999999999,"5min":0.17699999999999999,"15min":0.17299999999999999},"max":{"1min":0.371,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.27700000000000002,"5min":0.27000000000000002,"15min":0.26600000000000001},"min":{"1min":0.19900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.42799999999999999,"5min":1.153,"15min":1.153},"last":0.25600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359740698,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:11:14 2023","interfaces":[{"interface":"back","average":{"1min":0.20699999999999999,"5min":0.19,"15min":0.19600000000000001},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.13},"max":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.41099999999999998},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.309,"5min":0.26700000000000002,"15min":0.23999999999999999},"min":{"1min":0.17399999999999999,"5min":0.14799999999999999,"15min":0.14499999999999999},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.22700000000000001}]},{"osd":2,"last update":"Sun Dec 24 21:11:14 2023","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.28199999999999997,"15min":0.25900000000000001},"min":{"1min":0.184,"5min":0.17199999999999999,"15min":0.16800000000000001},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.314,"15min":0.28899999999999998},"min":{"1min":0.22,"5min":0.192,"15min":0.188},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.245}]}]},{"osd":1,"up_from":8,"seq":34359740701,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:11:15 2023","interfaces":[{"interface":"back","average":{"1min":0.191,"5min":0.19400000000000001,"15min":0.19700000000000001},"min":{"1min":0.155,"5min":0.126,"15min":0.126},"max":{"1min":0.23200000000000001,"5min":0.36699999999999999,"15min":0.437},"last":0.155},{"interface":"front","average":{"1min":0.22,"5min":0.224,"15min":0.22600000000000001},"min":{"1min":0.16900000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.28000000000000003,"5min":0.39600000000000002,"15min":0.439},"last":0.19800000000000001}]},{"osd":2,"last update":"Sun Dec 24 21:11:15 2023","interfaces":[{"interface":"back","average":{"1min":0.23599999999999999,"5min":0.22800000000000001,"15min":0.23699999999999999},"min":{"1min":0.17999999999999999,"5min":0.16400000000000001,"15min":0.157},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.185},{"interface":"front","average":{"1min":0.27800000000000002,"5min":0.27000000000000002,"15min":0.26600000000000001},"min":{"1min":0.223,"5min":0.20699999999999999,"15min":0.16900000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.24099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:11:25.845 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:25.845 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:25.845 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:25.845 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:25.845 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:25.846 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:25.846 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:25.846 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:25.846 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:25.846 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:25.846 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:25.846 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:25.846 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:25.847 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:11:25.856 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:25.860+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:26.115 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:26.119+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:26.716 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:26.720+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 25 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:26.887 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:26.891+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:27.069 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:27.073+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:27.743 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:27.747+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 25 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:27.873 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:27.877+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:28.096 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:28.100+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:28.725 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:28.728+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 25 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:28.839 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:28.843+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:29.088 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:29.091+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:29.761 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:29.765+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 25 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:29.850 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:29.853+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:30.111 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:30.114+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:30.791 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:30.794+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 25 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:30.823 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:30.827+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:31.151 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:31.155+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:31.840 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:31.843+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 25 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:31.872 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:31.876+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:32.142 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:32.146+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:32.830 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:32.834+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:32.853 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:32.857+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 25 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:33.100 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:33.104+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:33.794 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:33.797+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:33.836 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:33.840+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 25 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:34.070 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:34.074+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:34.831 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:34.833+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 25 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:34.836 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:34.839+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:35.099 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:35.101+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:35.805 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:35.807+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:35.856 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:35.859+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 25 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:36.061 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:36.064+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:36.799 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:36.801+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:36.810 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:36.813+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 25 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:37.035 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:37.038+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:37.822 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:37.824+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 25 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:37.846 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:37.849+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:38.084 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:38.086+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:38.847 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:38.850+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 25 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:38.871 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:38.874+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:39.074 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:39.077+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:39.854 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:39.857+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 25 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:39.890 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:39.893+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:40.047 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:40.050+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:40.866 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:40.868+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 25 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:40.908 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:40.911+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:41.038 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:41.041+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:41.848 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:41.851+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 25 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:41.957 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:41.960+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:42.085 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:42.088+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:42.870 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:42.873+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 25 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:43.001 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:43.004+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:43.109 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:43.112+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:43.846 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:43.848+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 25 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:43.953 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:43.956+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:44.140 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:44.142+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:44.861 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:44.864+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 25 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:44.996 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:44.998+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:45.153 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:45.155+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:45.848 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:11:45.852 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:45.855+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 25 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:45.998 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:46.000+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:46.129 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:46.132+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:46.221 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:11:46.223 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:11:46.235 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":5981,"stamp":"2023-12-24T21:11:44.677543+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,2,2,8,10,518],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001754"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2827,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:33.675344+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T21:08:33.675344+0000","last_peered":"2023-12-24T21:08:33.675344+0000","last_clean":"2023-12-24T21:08:33.675344+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T21:08:33.675344+0000","last_undegraded":"2023-12-24T21:08:33.675344+0000","last_fullsized":"2023-12-24T21:08:33.675344+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3621,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:34.024281+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T21:08:34.024281+0000","last_peered":"2023-12-24T21:08:34.024281+0000","last_clean":"2023-12-24T21:08:34.024281+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T21:08:34.024281+0000","last_undegraded":"2023-12-24T21:08:34.024281+0000","last_fullsized":"2023-12-24T21:08:34.024281+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2984,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:34.391689+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T21:08:34.391689+0000","last_peered":"2023-12-24T21:08:34.391689+0000","last_clean":"2023-12-24T21:08:34.391689+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T21:08:34.391689+0000","last_undegraded":"2023-12-24T21:08:34.391689+0000","last_fullsized":"2023-12-24T21:08:34.391689+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10168,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:11:24.186584+0000","last_change":"2023-12-24T21:11:24.186584+0000","last_active":"2023-12-24T21:11:24.186584+0000","last_peered":"2023-12-24T21:11:24.186584+0000","last_clean":"2023-12-24T21:11:24.186584+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:11:24.186584+0000","last_undegraded":"2023-12-24T21:11:24.186584+0000","last_fullsized":"2023-12-24T21:11:24.186584+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3926,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:35.134823+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:08:35.134823+0000","last_peered":"2023-12-24T21:08:35.134823+0000","last_clean":"2023-12-24T21:08:35.134823+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:08:35.134823+0000","last_undegraded":"2023-12-24T21:08:35.134823+0000","last_fullsized":"2023-12-24T21:08:35.134823+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11206,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:35.505493+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T21:08:35.505493+0000","last_peered":"2023-12-24T21:08:35.505493+0000","last_clean":"2023-12-24T21:08:35.505493+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T21:08:35.505493+0000","last_undegraded":"2023-12-24T21:08:35.505493+0000","last_fullsized":"2023-12-24T21:08:35.505493+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":287,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:35.841800+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T21:08:35.841800+0000","last_peered":"2023-12-24T21:08:35.841800+0000","last_clean":"2023-12-24T21:08:35.841800+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T21:08:35.841800+0000","last_undegraded":"2023-12-24T21:08:35.841800+0000","last_fullsized":"2023-12-24T21:08:35.841800+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3532,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:36.195031+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T21:08:36.195031+0000","last_peered":"2023-12-24T21:08:36.195031+0000","last_clean":"2023-12-24T21:08:36.195031+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T21:08:36.195031+0000","last_undegraded":"2023-12-24T21:08:36.195031+0000","last_fullsized":"2023-12-24T21:08:36.195031+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10734,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:36.544069+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T21:08:36.544069+0000","last_peered":"2023-12-24T21:08:36.544069+0000","last_clean":"2023-12-24T21:08:36.544069+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T21:08:36.544069+0000","last_undegraded":"2023-12-24T21:08:36.544069+0000","last_fullsized":"2023-12-24T21:08:36.544069+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3510,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:08:36.929523+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:08:36.929523+0000","last_peered":"2023-12-24T21:08:36.929523+0000","last_clean":"2023-12-24T21:08:36.929523+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:08:36.929523+0000","last_undegraded":"2023-12-24T21:08:36.929523+0000","last_fullsized":"2023-12-24T21:08:36.929523+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 357s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12876,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:11:29.091959+0000","last_change":"2023-12-24T21:11:29.091959+0000","last_active":"2023-12-24T21:11:29.091959+0000","last_peered":"2023-12-24T21:11:29.091959+0000","last_clean":"2023-12-24T21:11:29.091959+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:11:29.091959+0000","last_undegraded":"2023-12-24T21:11:29.091959+0000","last_fullsized":"2023-12-24T21:11:29.091959+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3152,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:11:38.087995+0000","last_change":"2023-12-24T21:11:38.087995+0000","last_active":"2023-12-24T21:11:38.087995+0000","last_peered":"2023-12-24T21:11:38.087995+0000","last_clean":"2023-12-24T21:11:38.087995+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:11:38.087995+0000","last_undegraded":"2023-12-24T21:11:38.087995+0000","last_fullsized":"2023-12-24T21:11:38.087995+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3316,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:37.978133+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T21:08:37.978133+0000","last_peered":"2023-12-24T21:08:37.978133+0000","last_clean":"2023-12-24T21:08:37.978133+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T21:08:37.978133+0000","last_undegraded":"2023-12-24T21:08:37.978133+0000","last_fullsized":"2023-12-24T21:08:37.978133+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740703,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,2,2,8,10,6],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:11:06 2023","interfaces":[{"interface":"back","average":{"1min":0.23499999999999999,"5min":0.23000000000000001,"15min":0.222},"min":{"1min":0.16800000000000001,"5min":0.14899999999999999,"15min":0.14399999999999999},"max":{"1min":0.44,"5min":1.198,"15min":1.198},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.249,"5min":0.245,"15min":0.24399999999999999},"min":{"1min":0.182,"5min":0.16900000000000001,"15min":0.16500000000000001},"max":{"1min":0.39400000000000002,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.22700000000000001}]},{"osd":1,"last update":"Sun Dec 24 21:11:06 2023","interfaces":[{"interface":"back","average":{"1min":0.254,"5min":0.252,"15min":0.24399999999999999},"min":{"1min":0.18099999999999999,"5min":0.17699999999999999,"15min":0.17299999999999999},"max":{"1min":0.371,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.248},{"interface":"front","average":{"1min":0.27700000000000002,"5min":0.27000000000000002,"15min":0.26600000000000001},"min":{"1min":0.19900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.42799999999999999,"5min":1.153,"15min":1.153},"last":0.26000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359740702,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:11:14 2023","interfaces":[{"interface":"back","average":{"1min":0.20699999999999999,"5min":0.19,"15min":0.19600000000000001},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.13},"max":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.41099999999999998},"last":0.13900000000000001},{"interface":"front","average":{"1min":0.309,"5min":0.26700000000000002,"15min":0.23999999999999999},"min":{"1min":0.17399999999999999,"5min":0.14799999999999999,"15min":0.14499999999999999},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.183}]},{"osd":2,"last update":"Sun Dec 24 21:11:14 2023","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.28199999999999997,"15min":0.25900000000000001},"min":{"1min":0.184,"5min":0.17199999999999999,"15min":0.16800000000000001},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.17399999999999999},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.314,"15min":0.28899999999999998},"min":{"1min":0.22,"5min":0.192,"15min":0.188},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.20300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359740705,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:11:15 2023","interfaces":[{"interface":"back","average":{"1min":0.191,"5min":0.19400000000000001,"15min":0.19700000000000001},"min":{"1min":0.155,"5min":0.126,"15min":0.126},"max":{"1min":0.23200000000000001,"5min":0.36699999999999999,"15min":0.437},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.22,"5min":0.224,"15min":0.22600000000000001},"min":{"1min":0.16900000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.28000000000000003,"5min":0.39600000000000002,"15min":0.439},"last":0.23999999999999999}]},{"osd":2,"last update":"Sun Dec 24 21:11:15 2023","interfaces":[{"interface":"back","average":{"1min":0.23599999999999999,"5min":0.22800000000000001,"15min":0.23699999999999999},"min":{"1min":0.17999999999999999,"5min":0.16400000000000001,"15min":0.157},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.214},{"interface":"front","average":{"1min":0.27800000000000002,"5min":0.27000000000000002,"15min":0.26600000000000001},"min":{"1min":0.223,"5min":0.20699999999999999,"15min":0.16900000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.27400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:11:46.237 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:46.237 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:46.237 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:46.237 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:46.238 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:46.238 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:46.238 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:46.238 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:46.238 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:46.239 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:46.239 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:46.239 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:46.239 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:11:46.239 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:11:46.803 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:46.805+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 25 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:46.961 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:46.964+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:47.155 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:47.158+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:47.802 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:47.804+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 25 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:47.945 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:47.947+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:48.188 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:48.191+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:48.774 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:48.777+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 25 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:48.944 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:48.947+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:49.216 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:49.218+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:49.818 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:49.820+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 25 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:49.906 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:49.908+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:50.180 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:50.183+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:50.836 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:50.838+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 25 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:50.939 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:50.942+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:51.228 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:51.230+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:51.843 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:51.845+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 25 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:51.971 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:51.974+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:52.186 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:52.189+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:52.822 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:52.825+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 25 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:52.946 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:52.949+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:53.215 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:53.218+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:53.795 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:53.797+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 25 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:53.938 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:53.940+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:54.191 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:54.193+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:54.820 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:54.823+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 25 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:54.938 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:54.941+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:55.162 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:55.164+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:55.808 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:55.810+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 25 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:55.940 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:55.943+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:56.113 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:56.115+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:56.770 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:56.773+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:56.981 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:56.984+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:57.101 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:57.104+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:57.764 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:57.767+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:58.022 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:58.024+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:58.141 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:58.144+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:58.749 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:58.751+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:59.012 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:59.015+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:59.122 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:11:59.124+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:59.773 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:11:59.775+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:11:59.970 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:11:59.972+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:00.147 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:00.149+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:00.775 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:00.778+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:01.005 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:01.007+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:01.104 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:01.106+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:01.732 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:01.735+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:01.994 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:01.997+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:02.107 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:02.110+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:02.723 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:02.726+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:02.977 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:02.980+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:03.078 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:03.080+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:03.716 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:03.719+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:04.027 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:04.030+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:04.054 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:04.057+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:04.703 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:04.706+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:05.010 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:05.012+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:05.080 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:05.083+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:05.673 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:05.676+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:06.050 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:06.054+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:06.070 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:06.074+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:06.240 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:12:06.578 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:12:06.581 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:12:06.593 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":5991,"stamp":"2023-12-24T21:12:04.680596+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,2,2,8,10,518],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001980"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2827,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:33.675344+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T21:08:33.675344+0000","last_peered":"2023-12-24T21:08:33.675344+0000","last_clean":"2023-12-24T21:08:33.675344+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T21:08:33.675344+0000","last_undegraded":"2023-12-24T21:08:33.675344+0000","last_fullsized":"2023-12-24T21:08:33.675344+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3621,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:34.024281+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T21:08:34.024281+0000","last_peered":"2023-12-24T21:08:34.024281+0000","last_clean":"2023-12-24T21:08:34.024281+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T21:08:34.024281+0000","last_undegraded":"2023-12-24T21:08:34.024281+0000","last_fullsized":"2023-12-24T21:08:34.024281+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2984,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:34.391689+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T21:08:34.391689+0000","last_peered":"2023-12-24T21:08:34.391689+0000","last_clean":"2023-12-24T21:08:34.391689+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T21:08:34.391689+0000","last_undegraded":"2023-12-24T21:08:34.391689+0000","last_fullsized":"2023-12-24T21:08:34.391689+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10174,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:11:58.145110+0000","last_change":"2023-12-24T21:11:58.145110+0000","last_active":"2023-12-24T21:11:58.145110+0000","last_peered":"2023-12-24T21:11:58.145110+0000","last_clean":"2023-12-24T21:11:58.145110+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:11:58.145110+0000","last_undegraded":"2023-12-24T21:11:58.145110+0000","last_fullsized":"2023-12-24T21:11:58.145110+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3926,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:35.134823+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:08:35.134823+0000","last_peered":"2023-12-24T21:08:35.134823+0000","last_clean":"2023-12-24T21:08:35.134823+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:08:35.134823+0000","last_undegraded":"2023-12-24T21:08:35.134823+0000","last_fullsized":"2023-12-24T21:08:35.134823+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11206,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:35.505493+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T21:08:35.505493+0000","last_peered":"2023-12-24T21:08:35.505493+0000","last_clean":"2023-12-24T21:08:35.505493+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T21:08:35.505493+0000","last_undegraded":"2023-12-24T21:08:35.505493+0000","last_fullsized":"2023-12-24T21:08:35.505493+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":287,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:35.841800+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T21:08:35.841800+0000","last_peered":"2023-12-24T21:08:35.841800+0000","last_clean":"2023-12-24T21:08:35.841800+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T21:08:35.841800+0000","last_undegraded":"2023-12-24T21:08:35.841800+0000","last_fullsized":"2023-12-24T21:08:35.841800+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3532,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:36.195031+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T21:08:36.195031+0000","last_peered":"2023-12-24T21:08:36.195031+0000","last_clean":"2023-12-24T21:08:36.195031+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T21:08:36.195031+0000","last_undegraded":"2023-12-24T21:08:36.195031+0000","last_fullsized":"2023-12-24T21:08:36.195031+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10734,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:36.544069+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T21:08:36.544069+0000","last_peered":"2023-12-24T21:08:36.544069+0000","last_clean":"2023-12-24T21:08:36.544069+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T21:08:36.544069+0000","last_undegraded":"2023-12-24T21:08:36.544069+0000","last_fullsized":"2023-12-24T21:08:36.544069+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3510,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:08:36.929523+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:08:36.929523+0000","last_peered":"2023-12-24T21:08:36.929523+0000","last_clean":"2023-12-24T21:08:36.929523+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:08:36.929523+0000","last_undegraded":"2023-12-24T21:08:36.929523+0000","last_fullsized":"2023-12-24T21:08:36.929523+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 357s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12879,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:11:48.192137+0000","last_change":"2023-12-24T21:11:48.192137+0000","last_active":"2023-12-24T21:11:48.192137+0000","last_peered":"2023-12-24T21:11:48.192137+0000","last_clean":"2023-12-24T21:11:48.192137+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:11:48.192137+0000","last_undegraded":"2023-12-24T21:11:48.192137+0000","last_fullsized":"2023-12-24T21:11:48.192137+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3155,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:11:52.190253+0000","last_change":"2023-12-24T21:11:52.190253+0000","last_active":"2023-12-24T21:11:52.190253+0000","last_peered":"2023-12-24T21:11:52.190253+0000","last_clean":"2023-12-24T21:11:52.190253+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:11:52.190253+0000","last_undegraded":"2023-12-24T21:11:52.190253+0000","last_fullsized":"2023-12-24T21:11:52.190253+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3316,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:37.978133+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T21:08:37.978133+0000","last_peered":"2023-12-24T21:08:37.978133+0000","last_clean":"2023-12-24T21:08:37.978133+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T21:08:37.978133+0000","last_undegraded":"2023-12-24T21:08:37.978133+0000","last_fullsized":"2023-12-24T21:08:37.978133+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740707,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,2,2,8,10,6],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:11:06 2023","interfaces":[{"interface":"back","average":{"1min":0.23499999999999999,"5min":0.23000000000000001,"15min":0.222},"min":{"1min":0.16800000000000001,"5min":0.14899999999999999,"15min":0.14399999999999999},"max":{"1min":0.44,"5min":1.198,"15min":1.198},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.249,"5min":0.245,"15min":0.24399999999999999},"min":{"1min":0.182,"5min":0.16900000000000001,"15min":0.16500000000000001},"max":{"1min":0.39400000000000002,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.45300000000000001}]},{"osd":1,"last update":"Sun Dec 24 21:11:06 2023","interfaces":[{"interface":"back","average":{"1min":0.254,"5min":0.252,"15min":0.24399999999999999},"min":{"1min":0.18099999999999999,"5min":0.17699999999999999,"15min":0.17299999999999999},"max":{"1min":0.371,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.27700000000000002,"5min":0.27000000000000002,"15min":0.26600000000000001},"min":{"1min":0.19900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.42799999999999999,"5min":1.153,"15min":1.153},"last":0.51400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359740706,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:11:14 2023","interfaces":[{"interface":"back","average":{"1min":0.20699999999999999,"5min":0.19,"15min":0.19600000000000001},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.13},"max":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.41099999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":0.309,"5min":0.26700000000000002,"15min":0.23999999999999999},"min":{"1min":0.17399999999999999,"5min":0.14799999999999999,"15min":0.14499999999999999},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.20100000000000001}]},{"osd":2,"last update":"Sun Dec 24 21:11:14 2023","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.28199999999999997,"15min":0.25900000000000001},"min":{"1min":0.184,"5min":0.17199999999999999,"15min":0.16800000000000001},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.246},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.314,"15min":0.28899999999999998},"min":{"1min":0.22,"5min":0.192,"15min":0.188},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.27900000000000003}]}]},{"osd":1,"up_from":8,"seq":34359740709,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:11:15 2023","interfaces":[{"interface":"back","average":{"1min":0.191,"5min":0.19400000000000001,"15min":0.19700000000000001},"min":{"1min":0.155,"5min":0.126,"15min":0.126},"max":{"1min":0.23200000000000001,"5min":0.36699999999999999,"15min":0.437},"last":0.19800000000000001},{"interface":"front","average":{"1min":0.22,"5min":0.224,"15min":0.22600000000000001},"min":{"1min":0.16900000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.28000000000000003,"5min":0.39600000000000002,"15min":0.439},"last":0.22800000000000001}]},{"osd":2,"last update":"Sun Dec 24 21:11:15 2023","interfaces":[{"interface":"back","average":{"1min":0.23599999999999999,"5min":0.22800000000000001,"15min":0.23699999999999999},"min":{"1min":0.17999999999999999,"5min":0.16400000000000001,"15min":0.157},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.27800000000000002,"5min":0.27000000000000002,"15min":0.26600000000000001},"min":{"1min":0.223,"5min":0.20699999999999999,"15min":0.16900000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.26200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:12:06.594 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:06.594 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:06.594 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:06.594 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:06.595 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:06.595 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:06.595 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:06.595 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:06.595 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:06.595 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:06.595 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:06.595 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:06.596 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:06.596 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:12:06.714 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:06.717+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:07.078 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:07.082+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:07.091 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:07.095+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:07.737 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:07.740+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:08.114 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:08.117+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:08.132 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:08.136+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:08.699 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:08.703+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:09.078 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:09.082+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:09.115 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:09.118+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:09.731 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:09.734+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:10.042 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:10.046+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:10.148 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:10.151+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:10.730 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:10.733+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:11.081 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:11.084+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:11.102 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:11.106+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:11.743 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:11.746+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:12.081 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:12.084+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:12.147 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:12.150+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:12.713 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:12.716+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:13.063 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:13.066+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:13.158 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:13.161+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:13.750 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:13.753+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:14.091 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:14.095+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:14.128 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:14.132+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:14.790 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:14.793+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:15.096 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:15.099+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:15.155 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:15.158+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:15.769 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:15.772+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:16.091 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:16.094+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:16.202 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:16.205+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:16.818 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:16.821+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:17.141 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:17.144+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:17.245 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:17.248+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:17.827 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:17.831+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:18.182 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:18.185+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:18.292 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:18.295+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:18.804 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:18.807+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:19.168 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:19.170+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:19.272 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:19.275+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:19.773 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:19.775+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:20.159 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:20.161+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:20.295 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:20.297+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:20.736 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:20.739+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:21.115 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:21.117+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:21.309 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:21.311+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:21.746 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:21.748+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:22.102 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:22.105+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:22.348 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:22.350+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:22.733 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:22.735+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:23.133 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:23.135+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:23.305 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:23.307+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:23.702 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:23.704+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:24.095 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:24.097+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:24.308 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:24.311+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:24.666 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:24.669+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:25.072 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:25.074+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:25.285 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:25.287+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:25.668 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:25.670+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:26.067 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:26.069+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:26.329 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:26.331+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:26.598 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:12:26.627 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:26.630+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:26.937 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:12:26.939 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:12:26.949 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":6002,"stamp":"2023-12-24T21:12:26.684281+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,2,2,8,10,518],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002198"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2827,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:33.675344+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T21:08:33.675344+0000","last_peered":"2023-12-24T21:08:33.675344+0000","last_clean":"2023-12-24T21:08:33.675344+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T21:08:33.675344+0000","last_undegraded":"2023-12-24T21:08:33.675344+0000","last_fullsized":"2023-12-24T21:08:33.675344+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3621,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:34.024281+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T21:08:34.024281+0000","last_peered":"2023-12-24T21:08:34.024281+0000","last_clean":"2023-12-24T21:08:34.024281+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T21:08:34.024281+0000","last_undegraded":"2023-12-24T21:08:34.024281+0000","last_fullsized":"2023-12-24T21:08:34.024281+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2984,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:34.391689+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T21:08:34.391689+0000","last_peered":"2023-12-24T21:08:34.391689+0000","last_clean":"2023-12-24T21:08:34.391689+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T21:08:34.391689+0000","last_undegraded":"2023-12-24T21:08:34.391689+0000","last_fullsized":"2023-12-24T21:08:34.391689+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10180,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:17.248562+0000","last_change":"2023-12-24T21:12:17.248562+0000","last_active":"2023-12-24T21:12:17.248562+0000","last_peered":"2023-12-24T21:12:17.248562+0000","last_clean":"2023-12-24T21:12:17.248562+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:12:17.248562+0000","last_undegraded":"2023-12-24T21:12:17.248562+0000","last_fullsized":"2023-12-24T21:12:17.248562+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3926,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:35.134823+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:08:35.134823+0000","last_peered":"2023-12-24T21:08:35.134823+0000","last_clean":"2023-12-24T21:08:35.134823+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:08:35.134823+0000","last_undegraded":"2023-12-24T21:08:35.134823+0000","last_fullsized":"2023-12-24T21:08:35.134823+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11206,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:35.505493+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T21:08:35.505493+0000","last_peered":"2023-12-24T21:08:35.505493+0000","last_clean":"2023-12-24T21:08:35.505493+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T21:08:35.505493+0000","last_undegraded":"2023-12-24T21:08:35.505493+0000","last_fullsized":"2023-12-24T21:08:35.505493+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":287,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:35.841800+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T21:08:35.841800+0000","last_peered":"2023-12-24T21:08:35.841800+0000","last_clean":"2023-12-24T21:08:35.841800+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T21:08:35.841800+0000","last_undegraded":"2023-12-24T21:08:35.841800+0000","last_fullsized":"2023-12-24T21:08:35.841800+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3532,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:36.195031+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T21:08:36.195031+0000","last_peered":"2023-12-24T21:08:36.195031+0000","last_clean":"2023-12-24T21:08:36.195031+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T21:08:36.195031+0000","last_undegraded":"2023-12-24T21:08:36.195031+0000","last_fullsized":"2023-12-24T21:08:36.195031+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10734,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:36.544069+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T21:08:36.544069+0000","last_peered":"2023-12-24T21:08:36.544069+0000","last_clean":"2023-12-24T21:08:36.544069+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T21:08:36.544069+0000","last_undegraded":"2023-12-24T21:08:36.544069+0000","last_fullsized":"2023-12-24T21:08:36.544069+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3510,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:08:36.929523+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:08:36.929523+0000","last_peered":"2023-12-24T21:08:36.929523+0000","last_clean":"2023-12-24T21:08:36.929523+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:08:36.929523+0000","last_undegraded":"2023-12-24T21:08:36.929523+0000","last_fullsized":"2023-12-24T21:08:36.929523+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 357s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12888,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:18.295868+0000","last_change":"2023-12-24T21:12:18.295868+0000","last_active":"2023-12-24T21:12:18.295868+0000","last_peered":"2023-12-24T21:12:18.295868+0000","last_clean":"2023-12-24T21:12:18.295868+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:12:18.295868+0000","last_undegraded":"2023-12-24T21:12:18.295868+0000","last_fullsized":"2023-12-24T21:12:18.295868+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3164,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:20.298490+0000","last_change":"2023-12-24T21:12:20.298490+0000","last_active":"2023-12-24T21:12:20.298490+0000","last_peered":"2023-12-24T21:12:20.298490+0000","last_clean":"2023-12-24T21:12:20.298490+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:12:20.298490+0000","last_undegraded":"2023-12-24T21:12:20.298490+0000","last_fullsized":"2023-12-24T21:12:20.298490+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3316,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:37.978133+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T21:08:37.978133+0000","last_peered":"2023-12-24T21:08:37.978133+0000","last_clean":"2023-12-24T21:08:37.978133+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T21:08:37.978133+0000","last_undegraded":"2023-12-24T21:08:37.978133+0000","last_fullsized":"2023-12-24T21:08:37.978133+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740711,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,2,2,8,10,6],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:12:11 2023","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":0.23300000000000001,"15min":0.223},"min":{"1min":0.156,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":0.44500000000000001,"5min":1.198,"15min":1.198},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.252,"5min":0.246,"15min":0.24399999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.45300000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.32800000000000001}]},{"osd":1,"last update":"Sun Dec 24 21:12:11 2023","interfaces":[{"interface":"back","average":{"1min":0.26400000000000001,"5min":0.25600000000000001,"15min":0.246},"min":{"1min":0.17899999999999999,"5min":0.17699999999999999,"15min":0.17299999999999999},"max":{"1min":0.47799999999999998,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.308},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.27400000000000002,"15min":0.26700000000000002},"min":{"1min":0.187,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.51400000000000001,"5min":1.153,"15min":1.153},"last":0.28899999999999998}]}]},{"osd":0,"up_from":8,"seq":34359740710,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:12:17 2023","interfaces":[{"interface":"back","average":{"1min":0.24099999999999999,"5min":0.20100000000000001,"15min":0.19800000000000001},"min":{"1min":0.13900000000000001,"5min":0.13900000000000001,"15min":0.13},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.22700000000000001,"5min":0.26800000000000002,"15min":0.23999999999999999},"min":{"1min":0.183,"5min":0.14799999999999999,"15min":0.14499999999999999},"max":{"1min":0.32000000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.22}]},{"osd":2,"last update":"Sun Dec 24 21:12:17 2023","interfaces":[{"interface":"back","average":{"1min":0.28000000000000003,"5min":0.28799999999999998,"15min":0.26100000000000001},"min":{"1min":0.17399999999999999,"5min":0.17199999999999999,"15min":0.16800000000000001},"max":{"1min":0.96299999999999997,"5min":1.139,"15min":1.139},"last":0.246},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.31900000000000001,"15min":0.28999999999999998},"min":{"1min":0.20300000000000001,"5min":0.192,"15min":0.188},"max":{"1min":0.98099999999999998,"5min":1.155,"15min":1.155},"last":0.30199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359740713,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:12:20 2023","interfaces":[{"interface":"back","average":{"1min":0.19600000000000001,"5min":0.19400000000000001,"15min":0.19800000000000001},"min":{"1min":0.13300000000000001,"5min":0.13300000000000001,"15min":0.126},"max":{"1min":0.31900000000000001,"5min":0.36699999999999999,"15min":0.437},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.23200000000000001,"5min":0.22500000000000001,"15min":0.22700000000000001},"min":{"1min":0.17399999999999999,"5min":0.16900000000000001,"15min":0.153},"max":{"1min":0.35299999999999998,"5min":0.39600000000000002,"15min":0.439},"last":0.20200000000000001}]},{"osd":2,"last update":"Sun Dec 24 21:12:20 2023","interfaces":[{"interface":"back","average":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.23799999999999999},"min":{"1min":0.16800000000000001,"5min":0.16500000000000001,"15min":0.157},"max":{"1min":0.36699999999999999,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.19},{"interface":"front","average":{"1min":0.27200000000000002,"5min":0.27100000000000002,"15min":0.26800000000000002},"min":{"1min":0.21299999999999999,"5min":0.21199999999999999,"15min":0.16900000000000001},"max":{"1min":0.38600000000000001,"5min":0.626,"15min":0.626},"last":0.255}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:12:26.951 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:26.951 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:26.951 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:26.951 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:26.951 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:26.952 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:26.952 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:26.952 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:26.952 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:26.952 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:26.953 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:26.953 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:26.953 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:26.953 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:12:27.084 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:27.087+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:27.342 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:27.345+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:27.582 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:27.584+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:28.073 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:28.075+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:28.326 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:28.328+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:28.581 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:28.583+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:29.047 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:29.049+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:29.375 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:29.377+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:29.610 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:29.612+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:30.003 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:30.005+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:30.347 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:30.349+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:30.578 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:30.580+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:30.970 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:30.972+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:31.390 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:31.392+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:31.589 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:31.591+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:32.011 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:32.013+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:32.357 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:32.358+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:32.582 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:32.583+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:33.046 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:33.048+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:33.377 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:33.379+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:33.626 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:33.628+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:34.019 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:34.021+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:34.410 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:34.411+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:34.619 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:34.621+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:35.054 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:35.057+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:35.375 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:35.377+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:35.656 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:35.658+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:36.016 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:36.019+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:36.346 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:36.349+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:36.624 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:36.627+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:37.037 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:37.039+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:37.304 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:37.306+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:37.616 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:37.618+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:38.043 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:38.045+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:38.307 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:38.309+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:38.632 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:38.634+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:38.995 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:38.997+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:39.335 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:39.337+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:39.646 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:39.648+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:39.968 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:39.970+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:40.328 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:40.330+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:40.676 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:40.678+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:40.937 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:40.939+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:41.341 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:41.343+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:41.676 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:41.678+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:41.961 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:41.963+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:42.377 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:42.379+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:42.726 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:42.728+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:42.921 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:42.923+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:43.406 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:43.408+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:43.742 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:43.744+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:43.914 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:43.892+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:44.399 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:44.401+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:44.777 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:44.779+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:44.885 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:44.887+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:45.438 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:45.440+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:45.737 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:45.739+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:45.903 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:45.905+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:46.417 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:46.419+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:46.785 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:46.787+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:46.928 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:46.930+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:46.954 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:12:47.331 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:12:47.334 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:12:47.344 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":6012,"stamp":"2023-12-24T21:12:46.687207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,2,2,8,10,518],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001595"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2827,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:33.675344+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T21:08:33.675344+0000","last_peered":"2023-12-24T21:08:33.675344+0000","last_clean":"2023-12-24T21:08:33.675344+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T21:08:33.675344+0000","last_undegraded":"2023-12-24T21:08:33.675344+0000","last_fullsized":"2023-12-24T21:08:33.675344+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3621,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:34.024281+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T21:08:34.024281+0000","last_peered":"2023-12-24T21:08:34.024281+0000","last_clean":"2023-12-24T21:08:34.024281+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T21:08:34.024281+0000","last_undegraded":"2023-12-24T21:08:34.024281+0000","last_fullsized":"2023-12-24T21:08:34.024281+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2984,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:34.391689+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T21:08:34.391689+0000","last_peered":"2023-12-24T21:08:34.391689+0000","last_clean":"2023-12-24T21:08:34.391689+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T21:08:34.391689+0000","last_undegraded":"2023-12-24T21:08:34.391689+0000","last_fullsized":"2023-12-24T21:08:34.391689+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10189,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:38.310065+0000","last_change":"2023-12-24T21:12:38.310065+0000","last_active":"2023-12-24T21:12:38.310065+0000","last_peered":"2023-12-24T21:12:38.310065+0000","last_clean":"2023-12-24T21:12:38.310065+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:12:38.310065+0000","last_undegraded":"2023-12-24T21:12:38.310065+0000","last_fullsized":"2023-12-24T21:12:38.310065+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3926,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:35.134823+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:08:35.134823+0000","last_peered":"2023-12-24T21:08:35.134823+0000","last_clean":"2023-12-24T21:08:35.134823+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:08:35.134823+0000","last_undegraded":"2023-12-24T21:08:35.134823+0000","last_fullsized":"2023-12-24T21:08:35.134823+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11206,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:35.505493+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T21:08:35.505493+0000","last_peered":"2023-12-24T21:08:35.505493+0000","last_clean":"2023-12-24T21:08:35.505493+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T21:08:35.505493+0000","last_undegraded":"2023-12-24T21:08:35.505493+0000","last_fullsized":"2023-12-24T21:08:35.505493+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":287,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:35.841800+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T21:08:35.841800+0000","last_peered":"2023-12-24T21:08:35.841800+0000","last_clean":"2023-12-24T21:08:35.841800+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T21:08:35.841800+0000","last_undegraded":"2023-12-24T21:08:35.841800+0000","last_fullsized":"2023-12-24T21:08:35.841800+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3532,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:36.195031+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T21:08:36.195031+0000","last_peered":"2023-12-24T21:08:36.195031+0000","last_clean":"2023-12-24T21:08:36.195031+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T21:08:36.195031+0000","last_undegraded":"2023-12-24T21:08:36.195031+0000","last_fullsized":"2023-12-24T21:08:36.195031+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10734,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:36.544069+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T21:08:36.544069+0000","last_peered":"2023-12-24T21:08:36.544069+0000","last_clean":"2023-12-24T21:08:36.544069+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T21:08:36.544069+0000","last_undegraded":"2023-12-24T21:08:36.544069+0000","last_fullsized":"2023-12-24T21:08:36.544069+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3510,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:08:36.929523+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:08:36.929523+0000","last_peered":"2023-12-24T21:08:36.929523+0000","last_clean":"2023-12-24T21:08:36.929523+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:08:36.929523+0000","last_undegraded":"2023-12-24T21:08:36.929523+0000","last_fullsized":"2023-12-24T21:08:36.929523+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 357s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12897,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:39.338766+0000","last_change":"2023-12-24T21:12:39.338766+0000","last_active":"2023-12-24T21:12:39.338766+0000","last_peered":"2023-12-24T21:12:39.338766+0000","last_clean":"2023-12-24T21:12:39.338766+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:12:39.338766+0000","last_undegraded":"2023-12-24T21:12:39.338766+0000","last_fullsized":"2023-12-24T21:12:39.338766+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3173,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:40.331411+0000","last_change":"2023-12-24T21:12:40.331411+0000","last_active":"2023-12-24T21:12:40.331411+0000","last_peered":"2023-12-24T21:12:40.331411+0000","last_clean":"2023-12-24T21:12:40.331411+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:12:40.331411+0000","last_undegraded":"2023-12-24T21:12:40.331411+0000","last_fullsized":"2023-12-24T21:12:40.331411+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3316,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:08:37.978133+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T21:08:37.978133+0000","last_peered":"2023-12-24T21:08:37.978133+0000","last_clean":"2023-12-24T21:08:37.978133+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T21:08:37.978133+0000","last_undegraded":"2023-12-24T21:08:37.978133+0000","last_fullsized":"2023-12-24T21:08:37.978133+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740715,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,2,2,8,10,6],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:12:11 2023","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":0.23300000000000001,"15min":0.223},"min":{"1min":0.156,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":0.44500000000000001,"5min":1.198,"15min":1.198},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.252,"5min":0.246,"15min":0.24399999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.45300000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.19600000000000001}]},{"osd":1,"last update":"Sun Dec 24 21:12:11 2023","interfaces":[{"interface":"back","average":{"1min":0.26400000000000001,"5min":0.25600000000000001,"15min":0.246},"min":{"1min":0.17899999999999999,"5min":0.17699999999999999,"15min":0.17299999999999999},"max":{"1min":0.47799999999999998,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.27400000000000002,"15min":0.26700000000000002},"min":{"1min":0.187,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.51400000000000001,"5min":1.153,"15min":1.153},"last":0.23100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359740714,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:12:17 2023","interfaces":[{"interface":"back","average":{"1min":0.24099999999999999,"5min":0.20100000000000001,"15min":0.19800000000000001},"min":{"1min":0.13900000000000001,"5min":0.13900000000000001,"15min":0.13},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.19700000000000001},{"interface":"front","average":{"1min":0.22700000000000001,"5min":0.26800000000000002,"15min":0.23999999999999999},"min":{"1min":0.183,"5min":0.14799999999999999,"15min":0.14499999999999999},"max":{"1min":0.32000000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.21299999999999999}]},{"osd":2,"last update":"Sun Dec 24 21:12:17 2023","interfaces":[{"interface":"back","average":{"1min":0.28000000000000003,"5min":0.28799999999999998,"15min":0.26100000000000001},"min":{"1min":0.17399999999999999,"5min":0.17199999999999999,"15min":0.16800000000000001},"max":{"1min":0.96299999999999997,"5min":1.139,"15min":1.139},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.31900000000000001,"15min":0.28999999999999998},"min":{"1min":0.20300000000000001,"5min":0.192,"15min":0.188},"max":{"1min":0.98099999999999998,"5min":1.155,"15min":1.155},"last":0.251}]}]},{"osd":1,"up_from":8,"seq":34359740717,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:12:20 2023","interfaces":[{"interface":"back","average":{"1min":0.19600000000000001,"5min":0.19400000000000001,"15min":0.19800000000000001},"min":{"1min":0.13300000000000001,"5min":0.13300000000000001,"15min":0.126},"max":{"1min":0.31900000000000001,"5min":0.36699999999999999,"15min":0.437},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.23200000000000001,"5min":0.22500000000000001,"15min":0.22700000000000001},"min":{"1min":0.17399999999999999,"5min":0.16900000000000001,"15min":0.153},"max":{"1min":0.35299999999999998,"5min":0.39600000000000002,"15min":0.439},"last":0.249}]},{"osd":2,"last update":"Sun Dec 24 21:12:20 2023","interfaces":[{"interface":"back","average":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.23799999999999999},"min":{"1min":0.16800000000000001,"5min":0.16500000000000001,"15min":0.157},"max":{"1min":0.36699999999999999,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.27200000000000002,"5min":0.27100000000000002,"15min":0.26800000000000002},"min":{"1min":0.21299999999999999,"5min":0.21199999999999999,"15min":0.16900000000000001},"max":{"1min":0.38600000000000001,"5min":0.626,"15min":0.626},"last":0.28999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:12:47.345 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:47.346 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:47.346 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:47.346 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:47.346 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:47.346 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:47.346 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:47.346 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:47.347 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:47.347 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:47.347 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:47.347 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:47.347 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:12:47.347 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.7 2023-12-24T21:12:47.374 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:47.376+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:47.722 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.7 on osd.1 to deep-scrub 2023-12-24T21:12:47.733 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.6 2023-12-24T21:12:47.743 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:47.745+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:47.914 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:47.917+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:48.083 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.6 on osd.1 to deep-scrub 2023-12-24T21:12:48.095 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2023-12-24T21:12:48.389 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:48.391+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:48.477 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2023-12-24T21:12:48.488 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-12-24T21:12:48.707 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:48.708+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:48.865 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-12-24T21:12:48.871 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:48.873+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:48.876 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.0 2023-12-24T21:12:49.213 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.0 on osd.2 to deep-scrub 2023-12-24T21:12:49.224 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-12-24T21:12:49.432 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:49.434+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:49.571 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-12-24T21:12:49.583 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.0 2023-12-24T21:12:49.681 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:49.683+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:49.831 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:49.833+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:49.955 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 1.0 on osd.1 to deep-scrub 2023-12-24T21:12:49.967 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.3 2023-12-24T21:12:50.316 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.3 on osd.1 to deep-scrub 2023-12-24T21:12:50.325 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-12-24T21:12:50.463 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:50.465+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:50.692 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:50.694+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:50.699 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-12-24T21:12:50.711 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.1 2023-12-24T21:12:50.839 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:50.842+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:51.051 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.1 on osd.2 to deep-scrub 2023-12-24T21:12:51.064 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-12-24T21:12:51.443 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:51.446+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:51.445 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-12-24T21:12:51.457 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.2 2023-12-24T21:12:51.728 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:51.731+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:51.829 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:51.832+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:51.839 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.2 on osd.0 to deep-scrub 2023-12-24T21:12:51.851 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.4 2023-12-24T21:12:52.232 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.4 on osd.1 to deep-scrub 2023-12-24T21:12:52.242 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:12:52.437 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:52.439+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:52.696 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:52.699+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:52.853 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:52.856+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:53.422 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:53.424+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:53.683 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:53.686+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:53.895 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:53.897+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:54.396 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:54.399+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:54.701 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:54.704+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:54.881 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:54.884+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:55.361 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:55.364+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:55.740 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:55.743+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:55.881 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:55.884+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:56.406 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:56.409+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:56.767 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:56.768+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:56.882 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:56.885+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:57.391 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:57.394+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:57.742 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:57.745+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:57.925 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:57.928+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:58.433 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:58.436+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:58.768 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:58.770+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:58.898 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:58.900+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:59.421 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:12:59.424+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:59.767 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:12:59.770+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:12:59.886 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:12:59.889+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:00.408 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:00.410+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:00.719 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:00.721+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:00.855 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:00.857+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:01.451 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:01.454+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:01.678 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:01.680+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:01.881 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:01.884+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:02.457 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:02.460+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:02.723 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:02.725+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:02.927 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:02.930+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:03.462 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:03.465+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:03.697 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:03.700+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:03.967 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:03.968+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:04.421 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:04.423+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:04.731 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:04.733+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:04.973 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:04.975+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:05.434 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:05.436+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:05.720 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:05.721+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:06.011 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:06.013+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:06.421 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:06.422+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:06.747 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:06.749+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:07.025 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:07.027+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:07.387 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:07.389+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:07.759 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:07.761+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:08.017 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:08.019+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:08.389 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:08.391+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:08.797 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:08.798+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:09.028 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:09.029+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:09.384 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:09.386+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:09.811 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:09.813+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:10.031 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:10.032+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:10.379 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:10.381+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:10.846 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:10.848+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:11.036 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:11.037+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:11.335 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:11.335+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:11.803 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:11.805+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:12.065 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:12.067+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:12.244 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:13:12.287 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:12.289+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:12.610 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:13:12.611 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:13:12.623 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":6024,"stamp":"2023-12-24T21:13:10.690868+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,2,2,8,10,518],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001920"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2828,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:40.637354+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T21:10:40.637354+0000","last_peered":"2023-12-24T21:10:40.637354+0000","last_clean":"2023-12-24T21:10:40.637354+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T21:10:40.637354+0000","last_undegraded":"2023-12-24T21:10:40.637354+0000","last_fullsized":"2023-12-24T21:10:40.637354+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3622,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:41.020605+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T21:10:41.020605+0000","last_peered":"2023-12-24T21:10:41.020605+0000","last_clean":"2023-12-24T21:10:41.020605+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T21:10:41.020605+0000","last_undegraded":"2023-12-24T21:10:41.020605+0000","last_fullsized":"2023-12-24T21:10:41.020605+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2985,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:41.367362+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T21:10:41.367362+0000","last_peered":"2023-12-24T21:10:41.367362+0000","last_clean":"2023-12-24T21:10:41.367362+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T21:10:41.367362+0000","last_undegraded":"2023-12-24T21:10:41.367362+0000","last_fullsized":"2023-12-24T21:10:41.367362+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10199,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:13:04.424370+0000","last_change":"2023-12-24T21:13:04.424370+0000","last_active":"2023-12-24T21:13:04.424370+0000","last_peered":"2023-12-24T21:13:04.424370+0000","last_clean":"2023-12-24T21:13:04.424370+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:13:04.424370+0000","last_undegraded":"2023-12-24T21:13:04.424370+0000","last_fullsized":"2023-12-24T21:13:04.424370+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3927,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:42.099100+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:10:42.099100+0000","last_peered":"2023-12-24T21:10:42.099100+0000","last_clean":"2023-12-24T21:10:42.099100+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:10:42.099100+0000","last_undegraded":"2023-12-24T21:10:42.099100+0000","last_fullsized":"2023-12-24T21:10:42.099100+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11207,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:42.472554+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T21:10:42.472554+0000","last_peered":"2023-12-24T21:10:42.472554+0000","last_clean":"2023-12-24T21:10:42.472554+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T21:10:42.472554+0000","last_undegraded":"2023-12-24T21:10:42.472554+0000","last_fullsized":"2023-12-24T21:10:42.472554+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":288,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:42.853382+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T21:10:42.853382+0000","last_peered":"2023-12-24T21:10:42.853382+0000","last_clean":"2023-12-24T21:10:42.853382+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T21:10:42.853382+0000","last_undegraded":"2023-12-24T21:10:42.853382+0000","last_fullsized":"2023-12-24T21:10:42.853382+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3533,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:43.226645+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T21:10:43.226645+0000","last_peered":"2023-12-24T21:10:43.226645+0000","last_clean":"2023-12-24T21:10:43.226645+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T21:10:43.226645+0000","last_undegraded":"2023-12-24T21:10:43.226645+0000","last_fullsized":"2023-12-24T21:10:43.226645+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10735,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:43.596359+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T21:10:43.596359+0000","last_peered":"2023-12-24T21:10:43.596359+0000","last_clean":"2023-12-24T21:10:43.596359+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T21:10:43.596359+0000","last_undegraded":"2023-12-24T21:10:43.596359+0000","last_fullsized":"2023-12-24T21:10:43.596359+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3511,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:10:43.991150+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:10:43.991150+0000","last_peered":"2023-12-24T21:10:43.991150+0000","last_clean":"2023-12-24T21:10:43.991150+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:10:43.991150+0000","last_undegraded":"2023-12-24T21:10:43.991150+0000","last_fullsized":"2023-12-24T21:10:43.991150+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 484s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12904,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:58.435874+0000","last_change":"2023-12-24T21:12:58.435874+0000","last_active":"2023-12-24T21:12:58.435874+0000","last_peered":"2023-12-24T21:12:58.435874+0000","last_clean":"2023-12-24T21:12:58.435874+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:12:58.435874+0000","last_undegraded":"2023-12-24T21:12:58.435874+0000","last_fullsized":"2023-12-24T21:12:58.435874+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3180,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:59.424457+0000","last_change":"2023-12-24T21:12:59.424457+0000","last_active":"2023-12-24T21:12:59.424457+0000","last_peered":"2023-12-24T21:12:59.424457+0000","last_clean":"2023-12-24T21:12:59.424457+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:12:59.424457+0000","last_undegraded":"2023-12-24T21:12:59.424457+0000","last_fullsized":"2023-12-24T21:12:59.424457+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3317,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:45.107192+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T21:10:45.107192+0000","last_peered":"2023-12-24T21:10:45.107192+0000","last_clean":"2023-12-24T21:10:45.107192+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T21:10:45.107192+0000","last_undegraded":"2023-12-24T21:10:45.107192+0000","last_fullsized":"2023-12-24T21:10:45.107192+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740720,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,2,2,8,10,6],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:12:11 2023","interfaces":[{"interface":"back","average":{"1min":0.22800000000000001,"5min":0.23300000000000001,"15min":0.223},"min":{"1min":0.156,"5min":0.153,"15min":0.14399999999999999},"max":{"1min":0.44500000000000001,"5min":1.198,"15min":1.198},"last":0.16400000000000001},{"interface":"front","average":{"1min":0.252,"5min":0.246,"15min":0.24399999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.45300000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.23300000000000001}]},{"osd":1,"last update":"Sun Dec 24 21:12:11 2023","interfaces":[{"interface":"back","average":{"1min":0.26400000000000001,"5min":0.25600000000000001,"15min":0.246},"min":{"1min":0.17899999999999999,"5min":0.17699999999999999,"15min":0.17299999999999999},"max":{"1min":0.47799999999999998,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.188},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.27400000000000002,"15min":0.26700000000000002},"min":{"1min":0.187,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.51400000000000001,"5min":1.153,"15min":1.153},"last":0.20300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359740719,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:12:17 2023","interfaces":[{"interface":"back","average":{"1min":0.24099999999999999,"5min":0.20100000000000001,"15min":0.19800000000000001},"min":{"1min":0.13900000000000001,"5min":0.13900000000000001,"15min":0.13},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.19400000000000001},{"interface":"front","average":{"1min":0.22700000000000001,"5min":0.26800000000000002,"15min":0.23999999999999999},"min":{"1min":0.183,"5min":0.14799999999999999,"15min":0.14499999999999999},"max":{"1min":0.32000000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.222}]},{"osd":2,"last update":"Sun Dec 24 21:12:17 2023","interfaces":[{"interface":"back","average":{"1min":0.28000000000000003,"5min":0.28799999999999998,"15min":0.26100000000000001},"min":{"1min":0.17399999999999999,"5min":0.17199999999999999,"15min":0.16800000000000001},"max":{"1min":0.96299999999999997,"5min":1.139,"15min":1.139},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.31900000000000001,"15min":0.28999999999999998},"min":{"1min":0.20300000000000001,"5min":0.192,"15min":0.188},"max":{"1min":0.98099999999999998,"5min":1.155,"15min":1.155},"last":0.318}]}]},{"osd":1,"up_from":8,"seq":34359740722,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:12:20 2023","interfaces":[{"interface":"back","average":{"1min":0.19600000000000001,"5min":0.19400000000000001,"15min":0.19800000000000001},"min":{"1min":0.13300000000000001,"5min":0.13300000000000001,"15min":0.126},"max":{"1min":0.31900000000000001,"5min":0.36699999999999999,"15min":0.437},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.23200000000000001,"5min":0.22500000000000001,"15min":0.22700000000000001},"min":{"1min":0.17399999999999999,"5min":0.16900000000000001,"15min":0.153},"max":{"1min":0.35299999999999998,"5min":0.39600000000000002,"15min":0.439},"last":0.26600000000000001}]},{"osd":2,"last update":"Sun Dec 24 21:12:20 2023","interfaces":[{"interface":"back","average":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.23799999999999999},"min":{"1min":0.16800000000000001,"5min":0.16500000000000001,"15min":0.157},"max":{"1min":0.36699999999999999,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.27200000000000002,"5min":0.27100000000000002,"15min":0.26800000000000002},"min":{"1min":0.21299999999999999,"5min":0.21199999999999999,"15min":0.16900000000000001},"max":{"1min":0.38600000000000001,"5min":0.626,"15min":0.626},"last":0.307}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:13:12.625 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:12.626 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:12.626 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:12.626 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:12.626 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:12.627 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:12.627 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:12.627 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:12.627 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:12.627 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:12.628 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:12.628 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:12.628 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:12.628 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:13:12.840 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:12.841+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:13.067 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:13.069+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:13.334 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:13.336+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:13.858 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:13.859+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:14.033 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:14.035+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:14.381 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:14.383+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:14.858 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:14.860+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:15.037 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:15.039+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:15.392 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:15.394+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:15.833 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:15.835+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:16.087 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:16.089+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:16.393 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:16.394+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:16.856 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:16.858+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:17.106 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:17.108+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:17.417 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:17.419+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:17.878 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:17.880+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:18.145 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:18.146+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:18.464 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:18.465+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:18.868 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:18.869+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:19.171 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:19.173+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:19.463 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:19.465+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:19.886 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:19.887+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:20.182 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:20.183+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:20.418 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:20.419+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:20.876 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:20.877+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:21.177 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:21.179+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:21.407 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:21.409+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:21.878 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:21.879+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:22.180 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:22.181+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:22.439 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:22.441+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:22.850 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:22.852+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:23.165 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:23.167+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:23.426 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:23.428+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:23.834 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:23.835+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:24.179 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:24.181+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:24.457 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:24.459+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:24.801 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:24.802+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:25.160 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:25.161+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:25.483 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:25.485+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:25.824 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:25.826+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:26.153 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:26.155+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:26.513 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:26.515+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:26.836 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:26.838+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:27.120 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:27.122+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:27.548 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:27.550+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:27.832 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:27.833+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:28.096 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:28.098+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:28.571 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:28.573+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:28.797 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:28.799+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:29.053 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:29.055+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:29.601 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:29.603+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:29.762 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:29.763+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:30.027 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:30.028+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:30.651 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:30.653+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:30.752 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:30.754+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:31.057 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:31.059+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:31.636 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:31.637+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:31.717 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:31.719+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:32.063 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:32.065+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:32.595 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:32.597+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:32.629 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:13:32.725 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:32.727+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:32.997 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:13:32.999 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:13:33.011 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":6035,"stamp":"2023-12-24T21:13:32.694177+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,2,2,8,10,518],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001891"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2828,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:40.637354+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T21:10:40.637354+0000","last_peered":"2023-12-24T21:10:40.637354+0000","last_clean":"2023-12-24T21:10:40.637354+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T21:10:40.637354+0000","last_undegraded":"2023-12-24T21:10:40.637354+0000","last_fullsized":"2023-12-24T21:10:40.637354+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3622,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:41.020605+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T21:10:41.020605+0000","last_peered":"2023-12-24T21:10:41.020605+0000","last_clean":"2023-12-24T21:10:41.020605+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T21:10:41.020605+0000","last_undegraded":"2023-12-24T21:10:41.020605+0000","last_fullsized":"2023-12-24T21:10:41.020605+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2985,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:41.367362+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T21:10:41.367362+0000","last_peered":"2023-12-24T21:10:41.367362+0000","last_clean":"2023-12-24T21:10:41.367362+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T21:10:41.367362+0000","last_undegraded":"2023-12-24T21:10:41.367362+0000","last_fullsized":"2023-12-24T21:10:41.367362+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10205,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:13:28.574652+0000","last_change":"2023-12-24T21:13:28.574652+0000","last_active":"2023-12-24T21:13:28.574652+0000","last_peered":"2023-12-24T21:13:28.574652+0000","last_clean":"2023-12-24T21:13:28.574652+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:13:28.574652+0000","last_undegraded":"2023-12-24T21:13:28.574652+0000","last_fullsized":"2023-12-24T21:13:28.574652+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3927,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:42.099100+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:10:42.099100+0000","last_peered":"2023-12-24T21:10:42.099100+0000","last_clean":"2023-12-24T21:10:42.099100+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:10:42.099100+0000","last_undegraded":"2023-12-24T21:10:42.099100+0000","last_fullsized":"2023-12-24T21:10:42.099100+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11207,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:42.472554+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T21:10:42.472554+0000","last_peered":"2023-12-24T21:10:42.472554+0000","last_clean":"2023-12-24T21:10:42.472554+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T21:10:42.472554+0000","last_undegraded":"2023-12-24T21:10:42.472554+0000","last_fullsized":"2023-12-24T21:10:42.472554+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":288,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:42.853382+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T21:10:42.853382+0000","last_peered":"2023-12-24T21:10:42.853382+0000","last_clean":"2023-12-24T21:10:42.853382+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T21:10:42.853382+0000","last_undegraded":"2023-12-24T21:10:42.853382+0000","last_fullsized":"2023-12-24T21:10:42.853382+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3533,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:43.226645+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T21:10:43.226645+0000","last_peered":"2023-12-24T21:10:43.226645+0000","last_clean":"2023-12-24T21:10:43.226645+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T21:10:43.226645+0000","last_undegraded":"2023-12-24T21:10:43.226645+0000","last_fullsized":"2023-12-24T21:10:43.226645+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10735,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:43.596359+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T21:10:43.596359+0000","last_peered":"2023-12-24T21:10:43.596359+0000","last_clean":"2023-12-24T21:10:43.596359+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T21:10:43.596359+0000","last_undegraded":"2023-12-24T21:10:43.596359+0000","last_fullsized":"2023-12-24T21:10:43.596359+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3511,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:10:43.991150+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:10:43.991150+0000","last_peered":"2023-12-24T21:10:43.991150+0000","last_clean":"2023-12-24T21:10:43.991150+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:10:43.991150+0000","last_undegraded":"2023-12-24T21:10:43.991150+0000","last_fullsized":"2023-12-24T21:10:43.991150+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 484s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12913,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:13:31.638963+0000","last_change":"2023-12-24T21:13:31.638963+0000","last_active":"2023-12-24T21:13:31.638963+0000","last_peered":"2023-12-24T21:13:31.638963+0000","last_clean":"2023-12-24T21:13:31.638963+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:13:31.638963+0000","last_undegraded":"2023-12-24T21:13:31.638963+0000","last_fullsized":"2023-12-24T21:13:31.638963+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3186,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:13:22.441708+0000","last_change":"2023-12-24T21:13:22.441708+0000","last_active":"2023-12-24T21:13:22.441708+0000","last_peered":"2023-12-24T21:13:22.441708+0000","last_clean":"2023-12-24T21:13:22.441708+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:13:22.441708+0000","last_undegraded":"2023-12-24T21:13:22.441708+0000","last_fullsized":"2023-12-24T21:13:22.441708+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3317,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:45.107192+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T21:10:45.107192+0000","last_peered":"2023-12-24T21:10:45.107192+0000","last_clean":"2023-12-24T21:10:45.107192+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T21:10:45.107192+0000","last_undegraded":"2023-12-24T21:10:45.107192+0000","last_fullsized":"2023-12-24T21:10:45.107192+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740725,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,2,2,8,10,6],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:13:16 2023","interfaces":[{"interface":"back","average":{"1min":0.20999999999999999,"5min":0.224,"15min":0.222},"min":{"1min":0.16400000000000001,"5min":0.156,"15min":0.14399999999999999},"max":{"1min":0.36599999999999999,"5min":1.0229999999999999,"15min":1.198},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.245,"5min":0.248,"15min":0.24399999999999999},"min":{"1min":0.184,"5min":0.161,"15min":0.161},"max":{"1min":0.34999999999999998,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.22}]},{"osd":1,"last update":"Sun Dec 24 21:13:16 2023","interfaces":[{"interface":"back","average":{"1min":0.254,"5min":0.25900000000000001,"15min":0.247},"min":{"1min":0.184,"5min":0.17699999999999999,"15min":0.17299999999999999},"max":{"1min":0.38,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.254,"5min":0.27400000000000002,"15min":0.26600000000000001},"min":{"1min":0.20300000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.39000000000000001,"5min":1.153,"15min":1.153},"last":0.25700000000000001}]}]},{"osd":0,"up_from":8,"seq":34359740724,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:13:17 2023","interfaces":[{"interface":"back","average":{"1min":0.24199999999999999,"5min":0.21299999999999999,"15min":0.20100000000000001},"min":{"1min":0.14699999999999999,"5min":0.13900000000000001,"15min":0.13},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.25},{"interface":"front","average":{"1min":0.221,"5min":0.252,"15min":0.23999999999999999},"min":{"1min":0.16600000000000001,"5min":0.16200000000000001,"15min":0.14499999999999999},"max":{"1min":0.26600000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.23100000000000001}]},{"osd":2,"last update":"Sun Dec 24 21:13:17 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.28199999999999997,"15min":0.26300000000000001},"min":{"1min":0.183,"5min":0.17199999999999999,"15min":0.17100000000000001},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.315},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.311,"15min":0.29199999999999998},"min":{"1min":0.20399999999999999,"5min":0.192,"15min":0.188},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.28699999999999998}]}]},{"osd":1,"up_from":8,"seq":34359740727,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:13:22 2023","interfaces":[{"interface":"back","average":{"1min":0.192,"5min":0.192,"15min":0.19800000000000001},"min":{"1min":0.14499999999999999,"5min":0.13300000000000001,"15min":0.126},"max":{"1min":0.23699999999999999,"5min":0.31900000000000001,"15min":0.437},"last":0.16800000000000001},{"interface":"front","average":{"1min":0.22500000000000001,"5min":0.223,"15min":0.22700000000000001},"min":{"1min":0.17899999999999999,"5min":0.16900000000000001,"15min":0.153},"max":{"1min":0.26600000000000001,"5min":0.35299999999999998,"15min":0.439},"last":0.19700000000000001}]},{"osd":2,"last update":"Sun Dec 24 21:13:22 2023","interfaces":[{"interface":"back","average":{"1min":0.218,"5min":0.223,"15min":0.23599999999999999},"min":{"1min":0.17399999999999999,"5min":0.16500000000000001,"15min":0.157},"max":{"1min":0.309,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.26600000000000001,"5min":0.26900000000000002,"15min":0.26800000000000002},"min":{"1min":0.22800000000000001,"5min":0.21299999999999999,"15min":0.16900000000000001},"max":{"1min":0.307,"5min":0.626,"15min":0.626},"last":0.223}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:13:33.013 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:33.013 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:33.013 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:33.013 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:33.013 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:33.014 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:33.014 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:33.014 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:33.014 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:33.015 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:33.015 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:33.015 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:33.015 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:33.015 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:13:33.101 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:33.103+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:33.646 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:33.647+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:33.769 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:33.771+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:34.091 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:34.093+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:34.658 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:34.659+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:34.792 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:34.795+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:35.129 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:35.131+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:35.703 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:35.705+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:35.809 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:35.812+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:36.158 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:36.160+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:36.706 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:36.708+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:36.771 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:36.774+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:37.133 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:37.136+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:37.744 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:37.747+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:37.803 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:37.805+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:38.149 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:38.152+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:38.779 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:38.781+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:38.824 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:38.827+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:39.147 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:39.150+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:39.776 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:39.778+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:39.800 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:39.802+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:40.148 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:40.150+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:40.756 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:40.758+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:40.844 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:40.847+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:41.145 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:41.147+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:41.711 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:41.713+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:41.867 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:41.870+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:42.139 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:42.142+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:42.756 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:42.759+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:42.864 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:42.867+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:43.125 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:43.127+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:43.791 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:43.793+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:43.839 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:43.842+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:44.144 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:44.146+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:44.796 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:44.798+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:44.823 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:44.826+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:45.140 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:45.142+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:45.800 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:45.802+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:45.828 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:45.830+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:46.132 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:46.134+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:46.774 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:46.777+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:46.781 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:46.783+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:47.159 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:47.161+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:47.733 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:47.735+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:47.810 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:47.813+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:48.172 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:48.174+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:48.696 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:48.697+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:48.814 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:48.815+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:49.178 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:49.179+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:49.705 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:49.706+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:49.769 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:49.770+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:50.174 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:50.176+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:50.665 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:50.667+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:50.751 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:50.753+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:51.222 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:51.223+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:51.681 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:51.683+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:51.721 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:51.722+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:52.239 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:52.240+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:52.658 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:52.659+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:52.725 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:52.726+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:53.017 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:13:53.265 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:53.267+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:53.375 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:13:53.377 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:13:53.389 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":6045,"stamp":"2023-12-24T21:13:52.697245+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,2,2,8,10,518],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001688"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2828,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:40.637354+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T21:10:40.637354+0000","last_peered":"2023-12-24T21:10:40.637354+0000","last_clean":"2023-12-24T21:10:40.637354+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T21:10:40.637354+0000","last_undegraded":"2023-12-24T21:10:40.637354+0000","last_fullsized":"2023-12-24T21:10:40.637354+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3622,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:41.020605+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T21:10:41.020605+0000","last_peered":"2023-12-24T21:10:41.020605+0000","last_clean":"2023-12-24T21:10:41.020605+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T21:10:41.020605+0000","last_undegraded":"2023-12-24T21:10:41.020605+0000","last_fullsized":"2023-12-24T21:10:41.020605+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2985,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:41.367362+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T21:10:41.367362+0000","last_peered":"2023-12-24T21:10:41.367362+0000","last_clean":"2023-12-24T21:10:41.367362+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T21:10:41.367362+0000","last_undegraded":"2023-12-24T21:10:41.367362+0000","last_fullsized":"2023-12-24T21:10:41.367362+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10214,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:13:47.735365+0000","last_change":"2023-12-24T21:13:47.735365+0000","last_active":"2023-12-24T21:13:47.735365+0000","last_peered":"2023-12-24T21:13:47.735365+0000","last_clean":"2023-12-24T21:13:47.735365+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:13:47.735365+0000","last_undegraded":"2023-12-24T21:13:47.735365+0000","last_fullsized":"2023-12-24T21:13:47.735365+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3927,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:42.099100+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:10:42.099100+0000","last_peered":"2023-12-24T21:10:42.099100+0000","last_clean":"2023-12-24T21:10:42.099100+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:10:42.099100+0000","last_undegraded":"2023-12-24T21:10:42.099100+0000","last_fullsized":"2023-12-24T21:10:42.099100+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11207,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:42.472554+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T21:10:42.472554+0000","last_peered":"2023-12-24T21:10:42.472554+0000","last_clean":"2023-12-24T21:10:42.472554+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T21:10:42.472554+0000","last_undegraded":"2023-12-24T21:10:42.472554+0000","last_fullsized":"2023-12-24T21:10:42.472554+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":288,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:42.853382+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T21:10:42.853382+0000","last_peered":"2023-12-24T21:10:42.853382+0000","last_clean":"2023-12-24T21:10:42.853382+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T21:10:42.853382+0000","last_undegraded":"2023-12-24T21:10:42.853382+0000","last_fullsized":"2023-12-24T21:10:42.853382+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3533,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:43.226645+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T21:10:43.226645+0000","last_peered":"2023-12-24T21:10:43.226645+0000","last_clean":"2023-12-24T21:10:43.226645+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T21:10:43.226645+0000","last_undegraded":"2023-12-24T21:10:43.226645+0000","last_fullsized":"2023-12-24T21:10:43.226645+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10735,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:43.596359+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T21:10:43.596359+0000","last_peered":"2023-12-24T21:10:43.596359+0000","last_clean":"2023-12-24T21:10:43.596359+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T21:10:43.596359+0000","last_undegraded":"2023-12-24T21:10:43.596359+0000","last_fullsized":"2023-12-24T21:10:43.596359+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3511,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:10:43.991150+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:10:43.991150+0000","last_peered":"2023-12-24T21:10:43.991150+0000","last_clean":"2023-12-24T21:10:43.991150+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:10:43.991150+0000","last_undegraded":"2023-12-24T21:10:43.991150+0000","last_fullsized":"2023-12-24T21:10:43.991150+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 484s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12922,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:13:50.667984+0000","last_change":"2023-12-24T21:13:50.667984+0000","last_active":"2023-12-24T21:13:50.667984+0000","last_peered":"2023-12-24T21:13:50.667984+0000","last_clean":"2023-12-24T21:13:50.667984+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:13:50.667984+0000","last_undegraded":"2023-12-24T21:13:50.667984+0000","last_fullsized":"2023-12-24T21:13:50.667984+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3195,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:13:46.777112+0000","last_change":"2023-12-24T21:13:46.777112+0000","last_active":"2023-12-24T21:13:46.777112+0000","last_peered":"2023-12-24T21:13:46.777112+0000","last_clean":"2023-12-24T21:13:46.777112+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:13:46.777112+0000","last_undegraded":"2023-12-24T21:13:46.777112+0000","last_fullsized":"2023-12-24T21:13:46.777112+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3317,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:45.107192+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T21:10:45.107192+0000","last_peered":"2023-12-24T21:10:45.107192+0000","last_clean":"2023-12-24T21:10:45.107192+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T21:10:45.107192+0000","last_undegraded":"2023-12-24T21:10:45.107192+0000","last_fullsized":"2023-12-24T21:10:45.107192+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740729,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,2,2,8,10,6],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:13:16 2023","interfaces":[{"interface":"back","average":{"1min":0.20999999999999999,"5min":0.224,"15min":0.222},"min":{"1min":0.16400000000000001,"5min":0.156,"15min":0.14399999999999999},"max":{"1min":0.36599999999999999,"5min":1.0229999999999999,"15min":1.198},"last":0.17899999999999999},{"interface":"front","average":{"1min":0.245,"5min":0.248,"15min":0.24399999999999999},"min":{"1min":0.184,"5min":0.161,"15min":0.161},"max":{"1min":0.34999999999999998,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.19500000000000001}]},{"osd":1,"last update":"Sun Dec 24 21:13:16 2023","interfaces":[{"interface":"back","average":{"1min":0.254,"5min":0.25900000000000001,"15min":0.247},"min":{"1min":0.184,"5min":0.17699999999999999,"15min":0.17299999999999999},"max":{"1min":0.38,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.254,"5min":0.27400000000000002,"15min":0.26600000000000001},"min":{"1min":0.20300000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.39000000000000001,"5min":1.153,"15min":1.153},"last":0.247}]}]},{"osd":0,"up_from":8,"seq":34359740728,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:13:17 2023","interfaces":[{"interface":"back","average":{"1min":0.24199999999999999,"5min":0.21299999999999999,"15min":0.20100000000000001},"min":{"1min":0.14699999999999999,"5min":0.13900000000000001,"15min":0.13},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.16800000000000001},{"interface":"front","average":{"1min":0.221,"5min":0.252,"15min":0.23999999999999999},"min":{"1min":0.16600000000000001,"5min":0.16200000000000001,"15min":0.14499999999999999},"max":{"1min":0.26600000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.18099999999999999}]},{"osd":2,"last update":"Sun Dec 24 21:13:17 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.28199999999999997,"15min":0.26300000000000001},"min":{"1min":0.183,"5min":0.17199999999999999,"15min":0.17100000000000001},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.311,"15min":0.29199999999999998},"min":{"1min":0.20399999999999999,"5min":0.192,"15min":0.188},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.26800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359740731,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:13:22 2023","interfaces":[{"interface":"back","average":{"1min":0.192,"5min":0.192,"15min":0.19800000000000001},"min":{"1min":0.14499999999999999,"5min":0.13300000000000001,"15min":0.126},"max":{"1min":0.23699999999999999,"5min":0.31900000000000001,"15min":0.437},"last":0.155},{"interface":"front","average":{"1min":0.22500000000000001,"5min":0.223,"15min":0.22700000000000001},"min":{"1min":0.17899999999999999,"5min":0.16900000000000001,"15min":0.153},"max":{"1min":0.26600000000000001,"5min":0.35299999999999998,"15min":0.439},"last":0.187}]},{"osd":2,"last update":"Sun Dec 24 21:13:22 2023","interfaces":[{"interface":"back","average":{"1min":0.218,"5min":0.223,"15min":0.23599999999999999},"min":{"1min":0.17399999999999999,"5min":0.16500000000000001,"15min":0.157},"max":{"1min":0.309,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.26600000000000001,"5min":0.26900000000000002,"15min":0.26800000000000002},"min":{"1min":0.22800000000000001,"5min":0.21299999999999999,"15min":0.16900000000000001},"max":{"1min":0.307,"5min":0.626,"15min":0.626},"last":0.219}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:13:53.390 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:53.391 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:53.391 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:53.391 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:53.391 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:53.391 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:53.391 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:53.391 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:53.392 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:53.392 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:53.392 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:53.392 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:53.392 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:13:53.392 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:13:53.689 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:53.691+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:53.702 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:53.703+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:54.303 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:54.304+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:54.658 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:54.659+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:54.661 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:54.662+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:55.352 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:55.353+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:55.649 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:55.651+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:55.653 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:55.654+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:56.342 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:56.343+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:56.633 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:56.634+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:56.700 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:56.701+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:57.368 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:57.369+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:57.662 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:57.663+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:57.707 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:57.708+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:58.322 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:58.323+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:58.612 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:58.614+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:58.684 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:58.685+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:59.297 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:13:59.299+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:59.628 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:13:59.630+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:13:59.709 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:13:59.710+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:00.260 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:00.261+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:00.641 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:00.643+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:00.678 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:00.680+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:01.246 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:01.247+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:01.648 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:01.649+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:01.684 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:01.685+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:02.233 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:02.234+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:02.681 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:02.683+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:02.734 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:02.735+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:03.251 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:03.252+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:03.670 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:03.671+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:03.727 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:03.728+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:04.284 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:04.286+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:04.658 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:04.659+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:04.763 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:04.764+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:05.255 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:05.257+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:05.694 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:05.695+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:05.743 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:05.744+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:06.252 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:06.253+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:06.735 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:06.736+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:06.766 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:06.767+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:07.278 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:07.279+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:07.752 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:07.754+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:07.754 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:07.756+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:08.275 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:08.276+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:08.704 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:08.705+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:08.742 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:08.743+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:09.246 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:09.247+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:09.710 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:09.711+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:09.719 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:09.720+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:10.233 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:10.235+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:10.714 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:10.715+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:10.743 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:10.744+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:11.209 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:11.210+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:11.739 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:11.741+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:11.779 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:11.780+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:12.224 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:12.225+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:12.698 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:12.699+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:12.738 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:12.740+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:13.197 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:13.199+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:13.393 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:14:13.701 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:13.702+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:13.760 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:13.761+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:13.764 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:14:13.766 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:14:13.778 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":6055,"stamp":"2023-12-24T21:14:12.700113+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,4,8,10,518],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001705"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2828,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:40.637354+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T21:10:40.637354+0000","last_peered":"2023-12-24T21:10:40.637354+0000","last_clean":"2023-12-24T21:10:40.637354+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T21:10:40.637354+0000","last_undegraded":"2023-12-24T21:10:40.637354+0000","last_fullsized":"2023-12-24T21:10:40.637354+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3622,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:41.020605+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T21:10:41.020605+0000","last_peered":"2023-12-24T21:10:41.020605+0000","last_clean":"2023-12-24T21:10:41.020605+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T21:10:41.020605+0000","last_undegraded":"2023-12-24T21:10:41.020605+0000","last_fullsized":"2023-12-24T21:10:41.020605+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2985,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:41.367362+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T21:10:41.367362+0000","last_peered":"2023-12-24T21:10:41.367362+0000","last_clean":"2023-12-24T21:10:41.367362+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T21:10:41.367362+0000","last_undegraded":"2023-12-24T21:10:41.367362+0000","last_fullsized":"2023-12-24T21:10:41.367362+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10226,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:11.741875+0000","last_change":"2023-12-24T21:14:11.741875+0000","last_active":"2023-12-24T21:14:11.741875+0000","last_peered":"2023-12-24T21:14:11.741875+0000","last_clean":"2023-12-24T21:14:11.741875+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:14:11.741875+0000","last_undegraded":"2023-12-24T21:14:11.741875+0000","last_fullsized":"2023-12-24T21:14:11.741875+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3927,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:42.099100+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:10:42.099100+0000","last_peered":"2023-12-24T21:10:42.099100+0000","last_clean":"2023-12-24T21:10:42.099100+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:10:42.099100+0000","last_undegraded":"2023-12-24T21:10:42.099100+0000","last_fullsized":"2023-12-24T21:10:42.099100+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11207,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:42.472554+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T21:10:42.472554+0000","last_peered":"2023-12-24T21:10:42.472554+0000","last_clean":"2023-12-24T21:10:42.472554+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T21:10:42.472554+0000","last_undegraded":"2023-12-24T21:10:42.472554+0000","last_fullsized":"2023-12-24T21:10:42.472554+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":288,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:42.853382+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T21:10:42.853382+0000","last_peered":"2023-12-24T21:10:42.853382+0000","last_clean":"2023-12-24T21:10:42.853382+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T21:10:42.853382+0000","last_undegraded":"2023-12-24T21:10:42.853382+0000","last_fullsized":"2023-12-24T21:10:42.853382+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3533,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:43.226645+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T21:10:43.226645+0000","last_peered":"2023-12-24T21:10:43.226645+0000","last_clean":"2023-12-24T21:10:43.226645+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T21:10:43.226645+0000","last_undegraded":"2023-12-24T21:10:43.226645+0000","last_fullsized":"2023-12-24T21:10:43.226645+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10735,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:43.596359+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T21:10:43.596359+0000","last_peered":"2023-12-24T21:10:43.596359+0000","last_clean":"2023-12-24T21:10:43.596359+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T21:10:43.596359+0000","last_undegraded":"2023-12-24T21:10:43.596359+0000","last_fullsized":"2023-12-24T21:10:43.596359+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3511,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:10:43.991150+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:10:43.991150+0000","last_peered":"2023-12-24T21:10:43.991150+0000","last_clean":"2023-12-24T21:10:43.991150+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:10:43.991150+0000","last_undegraded":"2023-12-24T21:10:43.991150+0000","last_fullsized":"2023-12-24T21:10:43.991150+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 484s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12931,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:06.737336+0000","last_change":"2023-12-24T21:14:06.737336+0000","last_active":"2023-12-24T21:14:06.737336+0000","last_peered":"2023-12-24T21:14:06.737336+0000","last_clean":"2023-12-24T21:14:06.737336+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:14:06.737336+0000","last_undegraded":"2023-12-24T21:14:06.737336+0000","last_fullsized":"2023-12-24T21:14:06.737336+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3207,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:08.706586+0000","last_change":"2023-12-24T21:14:08.706586+0000","last_active":"2023-12-24T21:14:08.706586+0000","last_peered":"2023-12-24T21:14:08.706586+0000","last_clean":"2023-12-24T21:14:08.706586+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:14:08.706586+0000","last_undegraded":"2023-12-24T21:14:08.706586+0000","last_fullsized":"2023-12-24T21:14:08.706586+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3317,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:45.107192+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T21:10:45.107192+0000","last_peered":"2023-12-24T21:10:45.107192+0000","last_clean":"2023-12-24T21:10:45.107192+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T21:10:45.107192+0000","last_undegraded":"2023-12-24T21:10:45.107192+0000","last_fullsized":"2023-12-24T21:10:45.107192+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740733,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,4,8,10,6],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:13:16 2023","interfaces":[{"interface":"back","average":{"1min":0.20999999999999999,"5min":0.224,"15min":0.222},"min":{"1min":0.16400000000000001,"5min":0.156,"15min":0.14399999999999999},"max":{"1min":0.36599999999999999,"5min":1.0229999999999999,"15min":1.198},"last":0.185},{"interface":"front","average":{"1min":0.245,"5min":0.248,"15min":0.24399999999999999},"min":{"1min":0.184,"5min":0.161,"15min":0.161},"max":{"1min":0.34999999999999998,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.249}]},{"osd":1,"last update":"Sun Dec 24 21:13:16 2023","interfaces":[{"interface":"back","average":{"1min":0.254,"5min":0.25900000000000001,"15min":0.247},"min":{"1min":0.184,"5min":0.17699999999999999,"15min":0.17299999999999999},"max":{"1min":0.38,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.254,"5min":0.27400000000000002,"15min":0.26600000000000001},"min":{"1min":0.20300000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.39000000000000001,"5min":1.153,"15min":1.153},"last":0.218}]}]},{"osd":0,"up_from":8,"seq":34359740732,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:13:17 2023","interfaces":[{"interface":"back","average":{"1min":0.24199999999999999,"5min":0.21299999999999999,"15min":0.20100000000000001},"min":{"1min":0.14699999999999999,"5min":0.13900000000000001,"15min":0.13},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.221,"5min":0.252,"15min":0.23999999999999999},"min":{"1min":0.16600000000000001,"5min":0.16200000000000001,"15min":0.14499999999999999},"max":{"1min":0.26600000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.23699999999999999}]},{"osd":2,"last update":"Sun Dec 24 21:13:17 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.28199999999999997,"15min":0.26300000000000001},"min":{"1min":0.183,"5min":0.17199999999999999,"15min":0.17100000000000001},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.311,"15min":0.29199999999999998},"min":{"1min":0.20399999999999999,"5min":0.192,"15min":0.188},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.28899999999999998}]}]},{"osd":1,"up_from":8,"seq":34359740735,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:13:22 2023","interfaces":[{"interface":"back","average":{"1min":0.192,"5min":0.192,"15min":0.19800000000000001},"min":{"1min":0.14499999999999999,"5min":0.13300000000000001,"15min":0.126},"max":{"1min":0.23699999999999999,"5min":0.31900000000000001,"15min":0.437},"last":0.216},{"interface":"front","average":{"1min":0.22500000000000001,"5min":0.223,"15min":0.22700000000000001},"min":{"1min":0.17899999999999999,"5min":0.16900000000000001,"15min":0.153},"max":{"1min":0.26600000000000001,"5min":0.35299999999999998,"15min":0.439},"last":0.26200000000000001}]},{"osd":2,"last update":"Sun Dec 24 21:13:22 2023","interfaces":[{"interface":"back","average":{"1min":0.218,"5min":0.223,"15min":0.23599999999999999},"min":{"1min":0.17399999999999999,"5min":0.16500000000000001,"15min":0.157},"max":{"1min":0.309,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.26600000000000001,"5min":0.26900000000000002,"15min":0.26800000000000002},"min":{"1min":0.22800000000000001,"5min":0.21299999999999999,"15min":0.16900000000000001},"max":{"1min":0.307,"5min":0.626,"15min":0.626},"last":0.29599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:14:13.779 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:13.779 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:13.779 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:13.779 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:13.779 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:13.779 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:13.779 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:13.780 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:13.780 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:13.780 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:13.780 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:13.780 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:13.780 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:13.780 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:14:14.208 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:14.209+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:14.655 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:14.656+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:14.710 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:14.712+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:15.186 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:15.187+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:15.643 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:15.644+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:15.722 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:15.723+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 30 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:16.144 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:16.145+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:16.688 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:16.689+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:16.710 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:16.711+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:17.097 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:17.098+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:17.683 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:17.684+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:17.701 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:17.702+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:18.145 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:18.147+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:18.658 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:18.659+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:18.747 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:18.748+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:19.162 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:19.163+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:19.643 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:19.644+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:19.778 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:19.779+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:20.157 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:20.158+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:20.683 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:20.685+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:20.794 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:20.796+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:21.122 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:21.125+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:21.689 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:21.691+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:21.828 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:21.830+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:22.154 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:22.156+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:22.643 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:22.646+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:22.779 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:22.781+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:23.188 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:23.190+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:23.622 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:23.624+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:23.813 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:23.815+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:24.213 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:24.216+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:24.634 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:24.636+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:24.852 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:24.854+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:25.204 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:25.207+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:25.671 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:25.673+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:25.826 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:25.828+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:26.239 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:26.241+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:26.664 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:26.666+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:26.817 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:26.819+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:27.233 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:27.235+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:27.621 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:27.623+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:27.801 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:27.803+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:28.195 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:28.197+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:28.619 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:28.621+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:28.762 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:28.765+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:29.223 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:29.225+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:29.616 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:29.618+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:29.765 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:29.766+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:30.199 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:30.201+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:30.666 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:30.668+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:30.729 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:30.732+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:31.154 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:31.157+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:31.640 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:31.642+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:31.740 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:31.742+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:32.108 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:32.111+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:32.642 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:32.644+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:32.720 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:32.722+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:33.138 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:33.140+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:33.642 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:33.643+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:33.733 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:33.734+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:33.781 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:14:34.119 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:14:34.121 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:14:34.132 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":6065,"stamp":"2023-12-24T21:14:32.703175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,2,0,0,4,8,10,518],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001924"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2828,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:40.637354+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T21:10:40.637354+0000","last_peered":"2023-12-24T21:10:40.637354+0000","last_clean":"2023-12-24T21:10:40.637354+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T21:10:40.637354+0000","last_undegraded":"2023-12-24T21:10:40.637354+0000","last_fullsized":"2023-12-24T21:10:40.637354+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3622,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:41.020605+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T21:10:41.020605+0000","last_peered":"2023-12-24T21:10:41.020605+0000","last_clean":"2023-12-24T21:10:41.020605+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T21:10:41.020605+0000","last_undegraded":"2023-12-24T21:10:41.020605+0000","last_fullsized":"2023-12-24T21:10:41.020605+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2985,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:41.367362+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T21:10:41.367362+0000","last_peered":"2023-12-24T21:10:41.367362+0000","last_clean":"2023-12-24T21:10:41.367362+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T21:10:41.367362+0000","last_undegraded":"2023-12-24T21:10:41.367362+0000","last_fullsized":"2023-12-24T21:10:41.367362+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10232,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:29.618454+0000","last_change":"2023-12-24T21:14:29.618454+0000","last_active":"2023-12-24T21:14:29.618454+0000","last_peered":"2023-12-24T21:14:29.618454+0000","last_clean":"2023-12-24T21:14:29.618454+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:14:29.618454+0000","last_undegraded":"2023-12-24T21:14:29.618454+0000","last_fullsized":"2023-12-24T21:14:29.618454+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3927,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:42.099100+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:10:42.099100+0000","last_peered":"2023-12-24T21:10:42.099100+0000","last_clean":"2023-12-24T21:10:42.099100+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:10:42.099100+0000","last_undegraded":"2023-12-24T21:10:42.099100+0000","last_fullsized":"2023-12-24T21:10:42.099100+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11207,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:42.472554+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T21:10:42.472554+0000","last_peered":"2023-12-24T21:10:42.472554+0000","last_clean":"2023-12-24T21:10:42.472554+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T21:10:42.472554+0000","last_undegraded":"2023-12-24T21:10:42.472554+0000","last_fullsized":"2023-12-24T21:10:42.472554+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":288,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:42.853382+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T21:10:42.853382+0000","last_peered":"2023-12-24T21:10:42.853382+0000","last_clean":"2023-12-24T21:10:42.853382+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T21:10:42.853382+0000","last_undegraded":"2023-12-24T21:10:42.853382+0000","last_fullsized":"2023-12-24T21:10:42.853382+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3533,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:43.226645+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T21:10:43.226645+0000","last_peered":"2023-12-24T21:10:43.226645+0000","last_clean":"2023-12-24T21:10:43.226645+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T21:10:43.226645+0000","last_undegraded":"2023-12-24T21:10:43.226645+0000","last_fullsized":"2023-12-24T21:10:43.226645+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10735,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:43.596359+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T21:10:43.596359+0000","last_peered":"2023-12-24T21:10:43.596359+0000","last_clean":"2023-12-24T21:10:43.596359+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T21:10:43.596359+0000","last_undegraded":"2023-12-24T21:10:43.596359+0000","last_fullsized":"2023-12-24T21:10:43.596359+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3511,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:10:43.991150+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:10:43.991150+0000","last_peered":"2023-12-24T21:10:43.991150+0000","last_clean":"2023-12-24T21:10:43.991150+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:10:43.991150+0000","last_undegraded":"2023-12-24T21:10:43.991150+0000","last_fullsized":"2023-12-24T21:10:43.991150+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 484s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12937,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:21.691425+0000","last_change":"2023-12-24T21:14:21.691425+0000","last_active":"2023-12-24T21:14:21.691425+0000","last_peered":"2023-12-24T21:14:21.691425+0000","last_clean":"2023-12-24T21:14:21.691425+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:14:21.691425+0000","last_undegraded":"2023-12-24T21:14:21.691425+0000","last_fullsized":"2023-12-24T21:14:21.691425+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3213,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:26.666571+0000","last_change":"2023-12-24T21:14:26.666571+0000","last_active":"2023-12-24T21:14:26.666571+0000","last_peered":"2023-12-24T21:14:26.666571+0000","last_clean":"2023-12-24T21:14:26.666571+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:14:26.666571+0000","last_undegraded":"2023-12-24T21:14:26.666571+0000","last_fullsized":"2023-12-24T21:14:26.666571+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3317,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:45.107192+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T21:10:45.107192+0000","last_peered":"2023-12-24T21:10:45.107192+0000","last_clean":"2023-12-24T21:10:45.107192+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T21:10:45.107192+0000","last_undegraded":"2023-12-24T21:10:45.107192+0000","last_fullsized":"2023-12-24T21:10:45.107192+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740737,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,2,0,0,4,8,10,6],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:14:19 2023","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":0.219,"15min":0.219},"min":{"1min":0.154,"5min":0.154,"15min":0.14399999999999999},"max":{"1min":0.39700000000000002,"5min":0.44500000000000001,"15min":1.198},"last":0.19700000000000001},{"interface":"front","average":{"1min":0.24299999999999999,"5min":0.24399999999999999,"15min":0.24199999999999999},"min":{"1min":0.17899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":0.40999999999999998,"5min":0.45300000000000001,"15min":1.1160000000000001},"last":0.24199999999999999}]},{"osd":1,"last update":"Sun Dec 24 21:14:19 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.26200000000000001,"15min":0.249},"min":{"1min":0.19,"5min":0.17699999999999999,"15min":0.17299999999999999},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1639999999999999},"last":0.218},{"interface":"front","average":{"1min":0.26300000000000001,"5min":0.26800000000000002,"15min":0.26400000000000001},"min":{"1min":0.21099999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":1.153},"last":0.27900000000000003}]}]},{"osd":0,"up_from":8,"seq":34359740736,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:14:20 2023","interfaces":[{"interface":"back","average":{"1min":0.23499999999999999,"5min":0.223,"15min":0.20399999999999999},"min":{"1min":0.157,"5min":0.13900000000000001,"15min":0.13900000000000001},"max":{"1min":0.39000000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.23899999999999999,"5min":0.249,"15min":0.24199999999999999},"min":{"1min":0.17899999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.41399999999999998,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.191}]},{"osd":2,"last update":"Sun Dec 24 21:14:20 2023","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.28100000000000003,"15min":0.26500000000000001},"min":{"1min":0.19900000000000001,"5min":0.17199999999999999,"15min":0.17100000000000001},"max":{"1min":0.43099999999999999,"5min":1.141,"15min":1.141},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.31,"15min":0.29399999999999998},"min":{"1min":0.221,"5min":0.192,"15min":0.188},"max":{"1min":0.501,"5min":1.157,"15min":1.157},"last":0.26800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359740739,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.183,"5min":0.188,"15min":0.19500000000000001},"min":{"1min":0.153,"5min":0.13300000000000001,"15min":0.126},"max":{"1min":0.246,"5min":0.31900000000000001,"15min":0.437},"last":0.17299999999999999},{"interface":"front","average":{"1min":0.21299999999999999,"5min":0.22,"15min":0.22500000000000001},"min":{"1min":0.17799999999999999,"5min":0.16900000000000001,"15min":0.153},"max":{"1min":0.28199999999999997,"5min":0.35299999999999998,"15min":0.439},"last":0.218}]},{"osd":2,"last update":"Sun Dec 24 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.21199999999999999,"5min":0.22,"15min":0.23400000000000001},"min":{"1min":0.17100000000000001,"5min":0.16500000000000001,"15min":0.157},"max":{"1min":0.29899999999999999,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.189},{"interface":"front","average":{"1min":0.25900000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.16900000000000001},"max":{"1min":0.377,"5min":0.626,"15min":0.626},"last":0.26000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:14:34.133 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:34.133 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:34.133 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:34.134 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:34.134 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:34.134 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:34.134 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:34.134 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:34.134 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:34.135 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:34.135 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:34.135 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:34.135 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:34.135 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:14:34.151 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:34.152+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:34.598 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:34.599+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:34.685 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:34.686+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:35.116 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:35.117+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:35.549 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:35.550+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:35.728 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:35.729+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:36.141 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:36.142+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:36.552 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:36.553+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:36.749 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:36.750+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:37.123 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:37.124+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:37.594 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:37.595+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:37.757 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:37.758+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:38.143 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:38.144+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:38.598 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:38.599+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:38.801 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:38.802+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:39.114 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:39.115+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:39.595 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:39.596+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:39.843 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:39.843+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:40.122 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:40.123+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:40.585 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:40.586+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:40.821 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:40.823+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:41.083 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:41.084+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:41.551 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:41.552+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:41.787 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:41.788+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:42.104 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:42.106+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:42.589 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:42.590+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:42.755 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:42.756+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:43.072 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:43.073+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:43.544 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:43.545+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:43.784 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:43.785+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:44.103 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:44.105+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:44.515 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:44.516+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:44.793 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:44.794+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:45.066 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:45.067+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:45.477 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:45.478+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:45.745 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:45.746+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:46.092 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:46.093+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:46.432 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:46.433+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:46.742 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:46.742+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:47.113 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:47.114+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:47.432 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:47.433+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:47.778 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:47.778+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:48.094 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:48.095+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:48.396 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:48.397+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:48.798 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:48.798+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:49.123 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:49.124+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:49.379 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:49.380+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:49.803 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:49.803+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:50.118 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:50.118+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:50.381 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:50.381+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:50.829 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:50.829+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:51.165 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:51.166+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:51.342 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:51.343+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:51.811 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:51.811+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:52.151 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:52.152+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:52.335 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:52.336+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:52.771 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:52.772+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:53.177 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:53.178+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:53.385 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:53.386+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:53.769 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:53.770+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:54.136 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:14:54.194 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:54.195+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:54.386 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:54.386+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:54.461 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:14:54.462 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:14:54.473 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":6075,"stamp":"2023-12-24T21:14:52.705918+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,2,0,0,4,6,12,518],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001558"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2828,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:40.637354+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T21:10:40.637354+0000","last_peered":"2023-12-24T21:10:40.637354+0000","last_clean":"2023-12-24T21:10:40.637354+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T21:10:40.637354+0000","last_undegraded":"2023-12-24T21:10:40.637354+0000","last_fullsized":"2023-12-24T21:10:40.637354+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3622,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:41.020605+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T21:10:41.020605+0000","last_peered":"2023-12-24T21:10:41.020605+0000","last_clean":"2023-12-24T21:10:41.020605+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T21:10:41.020605+0000","last_undegraded":"2023-12-24T21:10:41.020605+0000","last_fullsized":"2023-12-24T21:10:41.020605+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2985,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:41.367362+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T21:10:41.367362+0000","last_peered":"2023-12-24T21:10:41.367362+0000","last_clean":"2023-12-24T21:10:41.367362+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T21:10:41.367362+0000","last_undegraded":"2023-12-24T21:10:41.367362+0000","last_fullsized":"2023-12-24T21:10:41.367362+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10238,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:47.434648+0000","last_change":"2023-12-24T21:14:47.434648+0000","last_active":"2023-12-24T21:14:47.434648+0000","last_peered":"2023-12-24T21:14:47.434648+0000","last_clean":"2023-12-24T21:14:47.434648+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:14:47.434648+0000","last_undegraded":"2023-12-24T21:14:47.434648+0000","last_fullsized":"2023-12-24T21:14:47.434648+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3927,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:42.099100+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:10:42.099100+0000","last_peered":"2023-12-24T21:10:42.099100+0000","last_clean":"2023-12-24T21:10:42.099100+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:10:42.099100+0000","last_undegraded":"2023-12-24T21:10:42.099100+0000","last_fullsized":"2023-12-24T21:10:42.099100+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11207,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:42.472554+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T21:10:42.472554+0000","last_peered":"2023-12-24T21:10:42.472554+0000","last_clean":"2023-12-24T21:10:42.472554+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T21:10:42.472554+0000","last_undegraded":"2023-12-24T21:10:42.472554+0000","last_fullsized":"2023-12-24T21:10:42.472554+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":288,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:42.853382+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T21:10:42.853382+0000","last_peered":"2023-12-24T21:10:42.853382+0000","last_clean":"2023-12-24T21:10:42.853382+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T21:10:42.853382+0000","last_undegraded":"2023-12-24T21:10:42.853382+0000","last_fullsized":"2023-12-24T21:10:42.853382+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3533,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:43.226645+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T21:10:43.226645+0000","last_peered":"2023-12-24T21:10:43.226645+0000","last_clean":"2023-12-24T21:10:43.226645+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T21:10:43.226645+0000","last_undegraded":"2023-12-24T21:10:43.226645+0000","last_fullsized":"2023-12-24T21:10:43.226645+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10735,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:43.596359+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T21:10:43.596359+0000","last_peered":"2023-12-24T21:10:43.596359+0000","last_clean":"2023-12-24T21:10:43.596359+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T21:10:43.596359+0000","last_undegraded":"2023-12-24T21:10:43.596359+0000","last_fullsized":"2023-12-24T21:10:43.596359+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3511,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:10:43.991150+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:10:43.991150+0000","last_peered":"2023-12-24T21:10:43.991150+0000","last_clean":"2023-12-24T21:10:43.991150+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:10:43.991150+0000","last_undegraded":"2023-12-24T21:10:43.991150+0000","last_fullsized":"2023-12-24T21:10:43.991150+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 484s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12943,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:44.516913+0000","last_change":"2023-12-24T21:14:44.516913+0000","last_active":"2023-12-24T21:14:44.516913+0000","last_peered":"2023-12-24T21:14:44.516913+0000","last_clean":"2023-12-24T21:14:44.516913+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:14:44.516913+0000","last_undegraded":"2023-12-24T21:14:44.516913+0000","last_fullsized":"2023-12-24T21:14:44.516913+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3219,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:45.479003+0000","last_change":"2023-12-24T21:14:45.479003+0000","last_active":"2023-12-24T21:14:45.479003+0000","last_peered":"2023-12-24T21:14:45.479003+0000","last_clean":"2023-12-24T21:14:45.479003+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:14:45.479003+0000","last_undegraded":"2023-12-24T21:14:45.479003+0000","last_fullsized":"2023-12-24T21:14:45.479003+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3317,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:10:45.107192+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T21:10:45.107192+0000","last_peered":"2023-12-24T21:10:45.107192+0000","last_clean":"2023-12-24T21:10:45.107192+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T21:10:45.107192+0000","last_undegraded":"2023-12-24T21:10:45.107192+0000","last_fullsized":"2023-12-24T21:10:45.107192+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740741,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,2,0,0,4,6,12,6],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:14:19 2023","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":0.219,"15min":0.219},"min":{"1min":0.154,"5min":0.154,"15min":0.14399999999999999},"max":{"1min":0.39700000000000002,"5min":0.44500000000000001,"15min":1.198},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.24299999999999999,"5min":0.24399999999999999,"15min":0.24199999999999999},"min":{"1min":0.17899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":0.40999999999999998,"5min":0.45300000000000001,"15min":1.1160000000000001},"last":0.182}]},{"osd":1,"last update":"Sun Dec 24 21:14:19 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.26200000000000001,"15min":0.249},"min":{"1min":0.19,"5min":0.17699999999999999,"15min":0.17299999999999999},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1639999999999999},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.26300000000000001,"5min":0.26800000000000002,"15min":0.26400000000000001},"min":{"1min":0.21099999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":1.153},"last":0.219}]}]},{"osd":0,"up_from":8,"seq":34359740740,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:14:20 2023","interfaces":[{"interface":"back","average":{"1min":0.23499999999999999,"5min":0.223,"15min":0.20399999999999999},"min":{"1min":0.157,"5min":0.13900000000000001,"15min":0.13900000000000001},"max":{"1min":0.39000000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.16900000000000001},{"interface":"front","average":{"1min":0.23899999999999999,"5min":0.249,"15min":0.24199999999999999},"min":{"1min":0.17899999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.41399999999999998,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.20899999999999999}]},{"osd":2,"last update":"Sun Dec 24 21:14:20 2023","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.28100000000000003,"15min":0.26500000000000001},"min":{"1min":0.19900000000000001,"5min":0.17199999999999999,"15min":0.17100000000000001},"max":{"1min":0.43099999999999999,"5min":1.141,"15min":1.141},"last":0.20200000000000001},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.31,"15min":0.29399999999999998},"min":{"1min":0.221,"5min":0.192,"15min":0.188},"max":{"1min":0.501,"5min":1.157,"15min":1.157},"last":0.22900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359740743,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.183,"5min":0.188,"15min":0.19500000000000001},"min":{"1min":0.153,"5min":0.13300000000000001,"15min":0.126},"max":{"1min":0.246,"5min":0.31900000000000001,"15min":0.437},"last":0.151},{"interface":"front","average":{"1min":0.21299999999999999,"5min":0.22,"15min":0.22500000000000001},"min":{"1min":0.17799999999999999,"5min":0.16900000000000001,"15min":0.153},"max":{"1min":0.28199999999999997,"5min":0.35299999999999998,"15min":0.439},"last":0.17699999999999999}]},{"osd":2,"last update":"Sun Dec 24 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.21199999999999999,"5min":0.22,"15min":0.23400000000000001},"min":{"1min":0.17100000000000001,"5min":0.16500000000000001,"15min":0.157},"max":{"1min":0.29899999999999999,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.189},{"interface":"front","average":{"1min":0.25900000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.16900000000000001},"max":{"1min":0.377,"5min":0.626,"15min":0.626},"last":0.24299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:14:54.474 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:54.474 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:54.474 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:54.474 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:54.475 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:54.475 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:54.475 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:54.475 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:54.475 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:54.475 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:54.476 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:54.476 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:54.476 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:14:54.476 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.7 2023-12-24T21:14:54.796 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.7 on osd.1 to deep-scrub 2023-12-24T21:14:54.798 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:54.799+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:54.807 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.6 2023-12-24T21:14:55.175 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.6 on osd.1 to deep-scrub 2023-12-24T21:14:55.187 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2023-12-24T21:14:55.196 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:55.197+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:55.406 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:55.407+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:55.512 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2023-12-24T21:14:55.524 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-12-24T21:14:55.753 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:55.754+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:55.903 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-12-24T21:14:55.914 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.0 2023-12-24T21:14:56.156 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:56.157+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:56.296 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.0 on osd.2 to deep-scrub 2023-12-24T21:14:56.307 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-12-24T21:14:56.395 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:56.396+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:56.639 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-12-24T21:14:56.651 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.0 2023-12-24T21:14:56.716 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:56.717+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:57.036 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 1.0 on osd.1 to deep-scrub 2023-12-24T21:14:57.047 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.3 2023-12-24T21:14:57.143 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:57.144+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:57.375 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.3 on osd.1 to deep-scrub 2023-12-24T21:14:57.386 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-12-24T21:14:57.421 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:57.422+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:57.709 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-12-24T21:14:57.721 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.1 2023-12-24T21:14:57.758 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:57.759+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:58.079 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.1 on osd.2 to deep-scrub 2023-12-24T21:14:58.092 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-12-24T21:14:58.132 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:58.133+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:58.423 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-12-24T21:14:58.434 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.2 2023-12-24T21:14:58.455 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:58.456+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:58.776 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.2 on osd.0 to deep-scrub 2023-12-24T21:14:58.787 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.4 2023-12-24T21:14:58.801 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:58.803+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:59.091 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:14:59.092+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:59.111 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.4 on osd.1 to deep-scrub 2023-12-24T21:14:59.122 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:14:59.500 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:14:59.501+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:14:59.758 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:14:59.759+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:00.130 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:00.131+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:00.550 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:00.550+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:00.737 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:00.737+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:01.131 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:01.132+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:01.569 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:01.570+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:01.709 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:01.709+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:02.173 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:02.174+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:02.610 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:02.610+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:02.689 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:02.690+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:03.179 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:03.180+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:03.565 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:03.566+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:03.696 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:03.697+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:04.157 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:04.158+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:04.575 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:04.576+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:04.693 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:04.694+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:05.155 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:05.156+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:05.551 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:05.553+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:05.719 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:05.721+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:06.162 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:06.163+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:06.517 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:06.519+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:06.709 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:06.711+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:07.192 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:07.193+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:07.485 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:07.487+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:07.690 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:07.692+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:08.169 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:08.170+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:08.452 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:08.454+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:08.690 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:08.692+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:09.146 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:09.148+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:09.448 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:09.450+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:09.653 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:09.655+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:10.196 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:10.198+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:10.461 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:10.463+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:10.677 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:10.679+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:11.167 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:11.169+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:11.480 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:11.481+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:11.653 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:11.655+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:12.118 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:12.120+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:12.528 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:12.530+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:12.605 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:12.607+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:13.159 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:13.161+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:13.541 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:13.542+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:13.561 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:13.562+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:14.182 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:14.183+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:14.524 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:14.526+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:14.544 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:14.546+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:15.226 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:15.227+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:15.519 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:15.520+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:15.588 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:15.590+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:16.180 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:16.181+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:16.480 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:16.482+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:16.590 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:16.592+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:17.228 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:17.230+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:17.456 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:17.458+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:17.580 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:17.582+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:18.182 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:18.184+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:18.500 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:18.501+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:18.623 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:18.624+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:19.123 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:15:19.229 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:19.229+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:19.479 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:15:19.482 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:15:19.490 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:19.491+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:19.494 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":6088,"stamp":"2023-12-24T21:15:18.709958+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,2,0,0,4,6,12,518],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001877"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2829,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:47.722311+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T21:12:47.722311+0000","last_peered":"2023-12-24T21:12:47.722311+0000","last_clean":"2023-12-24T21:12:47.722311+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T21:12:47.722311+0000","last_undegraded":"2023-12-24T21:12:47.722311+0000","last_fullsized":"2023-12-24T21:12:47.722311+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3623,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:48.083686+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T21:12:48.083686+0000","last_peered":"2023-12-24T21:12:48.083686+0000","last_clean":"2023-12-24T21:12:48.083686+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T21:12:48.083686+0000","last_undegraded":"2023-12-24T21:12:48.083686+0000","last_fullsized":"2023-12-24T21:12:48.083686+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2986,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:48.477240+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T21:12:48.477240+0000","last_peered":"2023-12-24T21:12:48.477240+0000","last_clean":"2023-12-24T21:12:48.477240+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T21:12:48.477240+0000","last_undegraded":"2023-12-24T21:12:48.477240+0000","last_fullsized":"2023-12-24T21:12:48.477240+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10248,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:15:12.530467+0000","last_change":"2023-12-24T21:15:12.530467+0000","last_active":"2023-12-24T21:15:12.530467+0000","last_peered":"2023-12-24T21:15:12.530467+0000","last_clean":"2023-12-24T21:15:12.530467+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:15:12.530467+0000","last_undegraded":"2023-12-24T21:15:12.530467+0000","last_fullsized":"2023-12-24T21:15:12.530467+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3928,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:49.214434+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:12:49.214434+0000","last_peered":"2023-12-24T21:12:49.214434+0000","last_clean":"2023-12-24T21:12:49.214434+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:12:49.214434+0000","last_undegraded":"2023-12-24T21:12:49.214434+0000","last_fullsized":"2023-12-24T21:12:49.214434+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11208,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:49.571330+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T21:12:49.571330+0000","last_peered":"2023-12-24T21:12:49.571330+0000","last_clean":"2023-12-24T21:12:49.571330+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T21:12:49.571330+0000","last_undegraded":"2023-12-24T21:12:49.571330+0000","last_fullsized":"2023-12-24T21:12:49.571330+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":289,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:49.956385+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T21:12:49.956385+0000","last_peered":"2023-12-24T21:12:49.956385+0000","last_clean":"2023-12-24T21:12:49.956385+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T21:12:49.956385+0000","last_undegraded":"2023-12-24T21:12:49.956385+0000","last_fullsized":"2023-12-24T21:12:49.956385+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3534,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:50.314570+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T21:12:50.314570+0000","last_peered":"2023-12-24T21:12:50.314570+0000","last_clean":"2023-12-24T21:12:50.314570+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T21:12:50.314570+0000","last_undegraded":"2023-12-24T21:12:50.314570+0000","last_fullsized":"2023-12-24T21:12:50.314570+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10736,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:50.699675+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T21:12:50.699675+0000","last_peered":"2023-12-24T21:12:50.699675+0000","last_clean":"2023-12-24T21:12:50.699675+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T21:12:50.699675+0000","last_undegraded":"2023-12-24T21:12:50.699675+0000","last_fullsized":"2023-12-24T21:12:50.699675+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3512,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:12:51.052470+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:12:51.052470+0000","last_peered":"2023-12-24T21:12:51.052470+0000","last_clean":"2023-12-24T21:12:51.052470+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:12:51.052470+0000","last_undegraded":"2023-12-24T21:12:51.052470+0000","last_fullsized":"2023-12-24T21:12:51.052470+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 611s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12956,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:15:16.482373+0000","last_change":"2023-12-24T21:15:16.482373+0000","last_active":"2023-12-24T21:15:16.482373+0000","last_peered":"2023-12-24T21:15:16.482373+0000","last_clean":"2023-12-24T21:15:16.482373+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:15:16.482373+0000","last_undegraded":"2023-12-24T21:15:16.482373+0000","last_fullsized":"2023-12-24T21:15:16.482373+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3229,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:15:08.454275+0000","last_change":"2023-12-24T21:15:08.454275+0000","last_active":"2023-12-24T21:15:08.454275+0000","last_peered":"2023-12-24T21:15:08.454275+0000","last_clean":"2023-12-24T21:15:08.454275+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:15:08.454275+0000","last_undegraded":"2023-12-24T21:15:08.454275+0000","last_fullsized":"2023-12-24T21:15:08.454275+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3318,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:52.231439+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T21:12:52.231439+0000","last_peered":"2023-12-24T21:12:52.231439+0000","last_clean":"2023-12-24T21:12:52.231439+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T21:12:52.231439+0000","last_undegraded":"2023-12-24T21:12:52.231439+0000","last_fullsized":"2023-12-24T21:12:52.231439+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740746,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,2,0,0,4,6,12,6],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:14:19 2023","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":0.219,"15min":0.219},"min":{"1min":0.154,"5min":0.154,"15min":0.14399999999999999},"max":{"1min":0.39700000000000002,"5min":0.44500000000000001,"15min":1.198},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.24299999999999999,"5min":0.24399999999999999,"15min":0.24199999999999999},"min":{"1min":0.17899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":0.40999999999999998,"5min":0.45300000000000001,"15min":1.1160000000000001},"last":0.193}]},{"osd":1,"last update":"Sun Dec 24 21:14:19 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.26200000000000001,"15min":0.249},"min":{"1min":0.19,"5min":0.17699999999999999,"15min":0.17299999999999999},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1639999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.26300000000000001,"5min":0.26800000000000002,"15min":0.26400000000000001},"min":{"1min":0.21099999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":1.153},"last":0.23100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359740745,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:14:20 2023","interfaces":[{"interface":"back","average":{"1min":0.23499999999999999,"5min":0.223,"15min":0.20399999999999999},"min":{"1min":0.157,"5min":0.13900000000000001,"15min":0.13900000000000001},"max":{"1min":0.39000000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.17699999999999999},{"interface":"front","average":{"1min":0.23899999999999999,"5min":0.249,"15min":0.24199999999999999},"min":{"1min":0.17899999999999999,"5min":0.16200000000000001,"15min":0.14799999999999999},"max":{"1min":0.41399999999999998,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.20599999999999999}]},{"osd":2,"last update":"Sun Dec 24 21:14:20 2023","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.28100000000000003,"15min":0.26500000000000001},"min":{"1min":0.19900000000000001,"5min":0.17199999999999999,"15min":0.17100000000000001},"max":{"1min":0.43099999999999999,"5min":1.141,"15min":1.141},"last":0.221},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.31,"15min":0.29399999999999998},"min":{"1min":0.221,"5min":0.192,"15min":0.188},"max":{"1min":0.501,"5min":1.157,"15min":1.157},"last":0.251}]}]},{"osd":1,"up_from":8,"seq":34359740748,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.183,"5min":0.188,"15min":0.19500000000000001},"min":{"1min":0.153,"5min":0.13300000000000001,"15min":0.126},"max":{"1min":0.246,"5min":0.31900000000000001,"15min":0.437},"last":0.16600000000000001},{"interface":"front","average":{"1min":0.21299999999999999,"5min":0.22,"15min":0.22500000000000001},"min":{"1min":0.17799999999999999,"5min":0.16900000000000001,"15min":0.153},"max":{"1min":0.28199999999999997,"5min":0.35299999999999998,"15min":0.439},"last":0.20200000000000001}]},{"osd":2,"last update":"Sun Dec 24 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.21199999999999999,"5min":0.22,"15min":0.23400000000000001},"min":{"1min":0.17100000000000001,"5min":0.16500000000000001,"15min":0.157},"max":{"1min":0.29899999999999999,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.25900000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.16900000000000001},"max":{"1min":0.377,"5min":0.626,"15min":0.626},"last":0.24399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:15:19.496 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:15:19.496 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:15:19.496 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:15:19.496 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:15:19.497 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:15:19.497 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:15:19.497 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:15:19.497 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:15:19.497 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:15:19.498 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:15:19.498 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:15:19.498 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:15:19.498 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:15:19.499 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:15:19.661 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:19.662+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:20.213 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:20.213+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:20.531 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:20.532+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:20.612 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:20.612+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:21.224 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:21.224+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:21.536 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:21.537+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:21.642 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:21.642+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:22.252 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:22.253+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:22.569 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:22.570+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:22.647 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:22.648+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:23.280 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:23.281+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:23.526 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:23.527+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:23.646 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:23.646+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:24.234 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:24.234+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:24.560 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:24.561+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:24.662 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:24.662+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:25.214 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:25.214+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:25.610 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:25.611+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:25.675 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:25.676+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:26.248 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:26.249+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:26.566 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:26.567+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:26.670 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:26.670+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:27.245 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:27.246+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:27.521 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:27.521+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:27.668 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:27.669+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:28.219 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:28.219+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:28.490 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:28.490+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:28.629 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:28.630+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:29.186 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:29.186+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:29.446 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:29.447+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:29.618 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:29.618+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:30.164 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:30.165+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:30.465 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:30.465+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:30.632 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:30.633+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:31.168 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:31.168+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:31.462 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:31.463+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:31.662 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:31.662+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:32.167 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:32.168+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:32.476 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:32.477+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:32.693 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:32.694+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:33.171 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:33.172+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:33.473 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:33.474+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:33.674 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:33.675+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:34.136 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:34.137+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:34.494 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:34.494+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:34.720 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:34.720+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:35.185 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:35.186+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:35.500 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:35.501+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:35.692 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:35.693+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:36.210 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:36.210+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:36.455 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:36.456+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:36.674 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:36.674+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:37.248 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:37.249+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:37.425 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:37.426+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:37.700 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:37.701+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:38.253 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:38.254+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:38.417 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:38.418+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:38.740 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:38.741+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:39.249 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:39.249+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:39.420 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:39.421+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:39.499 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:15:39.745 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:39.746+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:40.051 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:15:40.054 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:15:40.066 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":6098,"stamp":"2023-12-24T21:15:38.712929+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,0,0,4,6,12,518],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001682"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2829,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:47.722311+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T21:12:47.722311+0000","last_peered":"2023-12-24T21:12:47.722311+0000","last_clean":"2023-12-24T21:12:47.722311+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T21:12:47.722311+0000","last_undegraded":"2023-12-24T21:12:47.722311+0000","last_fullsized":"2023-12-24T21:12:47.722311+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3623,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:48.083686+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T21:12:48.083686+0000","last_peered":"2023-12-24T21:12:48.083686+0000","last_clean":"2023-12-24T21:12:48.083686+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T21:12:48.083686+0000","last_undegraded":"2023-12-24T21:12:48.083686+0000","last_fullsized":"2023-12-24T21:12:48.083686+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2986,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:48.477240+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T21:12:48.477240+0000","last_peered":"2023-12-24T21:12:48.477240+0000","last_clean":"2023-12-24T21:12:48.477240+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T21:12:48.477240+0000","last_undegraded":"2023-12-24T21:12:48.477240+0000","last_fullsized":"2023-12-24T21:12:48.477240+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10254,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:15:32.477936+0000","last_change":"2023-12-24T21:15:32.477936+0000","last_active":"2023-12-24T21:15:32.477936+0000","last_peered":"2023-12-24T21:15:32.477936+0000","last_clean":"2023-12-24T21:15:32.477936+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:15:32.477936+0000","last_undegraded":"2023-12-24T21:15:32.477936+0000","last_fullsized":"2023-12-24T21:15:32.477936+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3928,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:49.214434+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:12:49.214434+0000","last_peered":"2023-12-24T21:12:49.214434+0000","last_clean":"2023-12-24T21:12:49.214434+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:12:49.214434+0000","last_undegraded":"2023-12-24T21:12:49.214434+0000","last_fullsized":"2023-12-24T21:12:49.214434+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11208,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:49.571330+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T21:12:49.571330+0000","last_peered":"2023-12-24T21:12:49.571330+0000","last_clean":"2023-12-24T21:12:49.571330+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T21:12:49.571330+0000","last_undegraded":"2023-12-24T21:12:49.571330+0000","last_fullsized":"2023-12-24T21:12:49.571330+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":289,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:49.956385+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T21:12:49.956385+0000","last_peered":"2023-12-24T21:12:49.956385+0000","last_clean":"2023-12-24T21:12:49.956385+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T21:12:49.956385+0000","last_undegraded":"2023-12-24T21:12:49.956385+0000","last_fullsized":"2023-12-24T21:12:49.956385+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3534,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:50.314570+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T21:12:50.314570+0000","last_peered":"2023-12-24T21:12:50.314570+0000","last_clean":"2023-12-24T21:12:50.314570+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T21:12:50.314570+0000","last_undegraded":"2023-12-24T21:12:50.314570+0000","last_fullsized":"2023-12-24T21:12:50.314570+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10736,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:50.699675+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T21:12:50.699675+0000","last_peered":"2023-12-24T21:12:50.699675+0000","last_clean":"2023-12-24T21:12:50.699675+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T21:12:50.699675+0000","last_undegraded":"2023-12-24T21:12:50.699675+0000","last_fullsized":"2023-12-24T21:12:50.699675+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3512,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:12:51.052470+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:12:51.052470+0000","last_peered":"2023-12-24T21:12:51.052470+0000","last_clean":"2023-12-24T21:12:51.052470+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:12:51.052470+0000","last_undegraded":"2023-12-24T21:12:51.052470+0000","last_fullsized":"2023-12-24T21:12:51.052470+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 611s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12962,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:15:35.502082+0000","last_change":"2023-12-24T21:15:35.502082+0000","last_active":"2023-12-24T21:15:35.502082+0000","last_peered":"2023-12-24T21:15:35.502082+0000","last_clean":"2023-12-24T21:15:35.502082+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:15:35.502082+0000","last_undegraded":"2023-12-24T21:15:35.502082+0000","last_fullsized":"2023-12-24T21:15:35.502082+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3235,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:15:27.522692+0000","last_change":"2023-12-24T21:15:27.522692+0000","last_active":"2023-12-24T21:15:27.522692+0000","last_peered":"2023-12-24T21:15:27.522692+0000","last_clean":"2023-12-24T21:15:27.522692+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:15:27.522692+0000","last_undegraded":"2023-12-24T21:15:27.522692+0000","last_fullsized":"2023-12-24T21:15:27.522692+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3318,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:52.231439+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T21:12:52.231439+0000","last_peered":"2023-12-24T21:12:52.231439+0000","last_clean":"2023-12-24T21:12:52.231439+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T21:12:52.231439+0000","last_undegraded":"2023-12-24T21:12:52.231439+0000","last_fullsized":"2023-12-24T21:12:52.231439+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740750,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,0,0,4,6,12,6],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:15:19 2023","interfaces":[{"interface":"back","average":{"1min":0.223,"5min":0.221,"15min":0.221},"min":{"1min":0.17100000000000001,"5min":0.154,"15min":0.14399999999999999},"max":{"1min":0.40500000000000003,"5min":0.44500000000000001,"15min":1.198},"last":0.187},{"interface":"front","average":{"1min":0.247,"5min":0.247,"15min":0.24399999999999999},"min":{"1min":0.182,"5min":0.161,"15min":0.161},"max":{"1min":0.42599999999999999,"5min":0.45300000000000001,"15min":1.1160000000000001},"last":0.19700000000000001}]},{"osd":1,"last update":"Sun Dec 24 21:15:19 2023","interfaces":[{"interface":"back","average":{"1min":0.26100000000000001,"5min":0.26500000000000001,"15min":0.252},"min":{"1min":0.189,"5min":0.17899999999999999,"15min":0.17699999999999999},"max":{"1min":0.49199999999999999,"5min":1.1499999999999999,"15min":1.1639999999999999},"last":0.17399999999999999},{"interface":"front","average":{"1min":0.28399999999999997,"5min":0.27100000000000002,"15min":0.26700000000000002},"min":{"1min":0.219,"5min":0.187,"15min":0.16900000000000001},"max":{"1min":0.46200000000000002,"5min":0.51400000000000001,"15min":1.153},"last":0.22600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359740749,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:15:22 2023","interfaces":[{"interface":"back","average":{"1min":0.246,"5min":0.23400000000000001,"15min":0.20699999999999999},"min":{"1min":0.158,"5min":0.13900000000000001,"15min":0.13900000000000001},"max":{"1min":0.71599999999999997,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.224,"5min":0.24399999999999999,"15min":0.24199999999999999},"min":{"1min":0.159,"5min":0.159,"15min":0.14799999999999999},"max":{"1min":0.41899999999999998,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.224}]},{"osd":2,"last update":"Sun Dec 24 21:15:22 2023","interfaces":[{"interface":"back","average":{"1min":0.246,"5min":0.27700000000000002,"15min":0.26500000000000001},"min":{"1min":0.191,"5min":0.17399999999999999,"15min":0.17100000000000001},"max":{"1min":0.39600000000000002,"5min":1.141,"15min":1.141},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.27900000000000003,"5min":0.30599999999999999,"15min":0.29499999999999998},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.188},"max":{"1min":0.45200000000000001,"5min":1.157,"15min":1.157},"last":0.245}]}]},{"osd":1,"up_from":8,"seq":34359740752,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.193,"5min":0.191,"15min":0.19500000000000001},"min":{"1min":0.151,"5min":0.13300000000000001,"15min":0.126},"max":{"1min":0.249,"5min":0.31900000000000001,"15min":0.437},"last":0.184},{"interface":"front","average":{"1min":0.219,"5min":0.221,"15min":0.22500000000000001},"min":{"1min":0.17699999999999999,"5min":0.16900000000000001,"15min":0.153},"max":{"1min":0.27500000000000002,"5min":0.35299999999999998,"15min":0.439},"last":0.219}]},{"osd":2,"last update":"Sun Dec 24 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.22,"5min":0.222,"15min":0.23300000000000001},"min":{"1min":0.17699999999999999,"5min":0.16800000000000001,"15min":0.157},"max":{"1min":0.30599999999999999,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.26200000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"min":{"1min":0.23200000000000001,"5min":0.20599999999999999,"15min":0.16900000000000001},"max":{"1min":0.32200000000000001,"5min":0.626,"15min":0.626},"last":0.26700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:15:40.067 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:15:40.067 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:15:40.067 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:15:40.068 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:15:40.068 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:15:40.068 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:15:40.068 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:15:40.068 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:15:40.068 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:15:40.068 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:15:40.069 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:15:40.069 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:15:40.069 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:15:40.069 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:15:40.282 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:40.283+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:40.396 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:40.397+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:40.771 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:40.772+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:41.308 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:41.309+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:41.434 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:41.435+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:41.760 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:41.761+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:42.296 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:42.296+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:42.460 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:42.461+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:42.725 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:42.725+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:43.248 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:43.249+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:43.488 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:43.488+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:43.741 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:43.741+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:44.226 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:44.227+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:44.485 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:44.486+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:44.774 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:44.774+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:45.199 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:45.199+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:45.530 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:45.530+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:45.765 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:45.765+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:46.180 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:46.181+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:46.516 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:46.517+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:46.814 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:46.815+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:47.162 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:47.163+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:47.508 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:47.509+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:47.856 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:47.858+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:48.190 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:48.191+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:48.510 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:48.511+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:48.906 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:48.906+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:49.179 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:49.179+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:49.462 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:49.463+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:49.948 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:49.949+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:50.194 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:50.196+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:50.445 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:50.447+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:50.977 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:50.978+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:51.226 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:51.227+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:51.409 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:51.410+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:51.961 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:51.962+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:52.200 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:52.201+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:52.392 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:52.393+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:52.933 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:52.934+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:53.200 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:53.202+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:53.409 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:53.411+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:53.945 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:53.946+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:54.202 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:54.204+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:54.453 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:54.454+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:54.943 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:54.944+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:55.181 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:55.183+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:55.472 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:55.473+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:55.928 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:55.929+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:56.138 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:56.139+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:56.490 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:56.492+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:56.964 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:56.965+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:57.177 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:57.179+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:57.511 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:57.512+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:58.013 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:58.014+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:58.133 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:58.135+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:58.471 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:58.472+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:58.980 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:58.982+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:59.103 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:15:59.105+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:59.425 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:15:59.427+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:15:59.986 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:15:59.987+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:00.069 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:16:00.130 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:00.131+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:00.398 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:00.399+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:00.432 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:16:00.435 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:16:00.447 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":6108,"stamp":"2023-12-24T21:15:58.715811+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,2,0,4,6,12,518],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001581"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2829,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:47.722311+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T21:12:47.722311+0000","last_peered":"2023-12-24T21:12:47.722311+0000","last_clean":"2023-12-24T21:12:47.722311+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T21:12:47.722311+0000","last_undegraded":"2023-12-24T21:12:47.722311+0000","last_fullsized":"2023-12-24T21:12:47.722311+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3623,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:48.083686+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T21:12:48.083686+0000","last_peered":"2023-12-24T21:12:48.083686+0000","last_clean":"2023-12-24T21:12:48.083686+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T21:12:48.083686+0000","last_undegraded":"2023-12-24T21:12:48.083686+0000","last_fullsized":"2023-12-24T21:12:48.083686+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2986,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:48.477240+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T21:12:48.477240+0000","last_peered":"2023-12-24T21:12:48.477240+0000","last_clean":"2023-12-24T21:12:48.477240+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T21:12:48.477240+0000","last_undegraded":"2023-12-24T21:12:48.477240+0000","last_fullsized":"2023-12-24T21:12:48.477240+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10260,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:15:49.464433+0000","last_change":"2023-12-24T21:15:49.464433+0000","last_active":"2023-12-24T21:15:49.464433+0000","last_peered":"2023-12-24T21:15:49.464433+0000","last_clean":"2023-12-24T21:15:49.464433+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:15:49.464433+0000","last_undegraded":"2023-12-24T21:15:49.464433+0000","last_fullsized":"2023-12-24T21:15:49.464433+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3928,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:49.214434+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:12:49.214434+0000","last_peered":"2023-12-24T21:12:49.214434+0000","last_clean":"2023-12-24T21:12:49.214434+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:12:49.214434+0000","last_undegraded":"2023-12-24T21:12:49.214434+0000","last_fullsized":"2023-12-24T21:12:49.214434+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11208,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:49.571330+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T21:12:49.571330+0000","last_peered":"2023-12-24T21:12:49.571330+0000","last_clean":"2023-12-24T21:12:49.571330+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T21:12:49.571330+0000","last_undegraded":"2023-12-24T21:12:49.571330+0000","last_fullsized":"2023-12-24T21:12:49.571330+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":289,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:49.956385+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T21:12:49.956385+0000","last_peered":"2023-12-24T21:12:49.956385+0000","last_clean":"2023-12-24T21:12:49.956385+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T21:12:49.956385+0000","last_undegraded":"2023-12-24T21:12:49.956385+0000","last_fullsized":"2023-12-24T21:12:49.956385+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3534,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:50.314570+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T21:12:50.314570+0000","last_peered":"2023-12-24T21:12:50.314570+0000","last_clean":"2023-12-24T21:12:50.314570+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T21:12:50.314570+0000","last_undegraded":"2023-12-24T21:12:50.314570+0000","last_fullsized":"2023-12-24T21:12:50.314570+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10736,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:50.699675+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T21:12:50.699675+0000","last_peered":"2023-12-24T21:12:50.699675+0000","last_clean":"2023-12-24T21:12:50.699675+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T21:12:50.699675+0000","last_undegraded":"2023-12-24T21:12:50.699675+0000","last_fullsized":"2023-12-24T21:12:50.699675+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3512,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:12:51.052470+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:12:51.052470+0000","last_peered":"2023-12-24T21:12:51.052470+0000","last_clean":"2023-12-24T21:12:51.052470+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:12:51.052470+0000","last_undegraded":"2023-12-24T21:12:51.052470+0000","last_fullsized":"2023-12-24T21:12:51.052470+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 611s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12968,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:15:54.454515+0000","last_change":"2023-12-24T21:15:54.454515+0000","last_active":"2023-12-24T21:15:54.454515+0000","last_peered":"2023-12-24T21:15:54.454515+0000","last_clean":"2023-12-24T21:15:54.454515+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:15:54.454515+0000","last_undegraded":"2023-12-24T21:15:54.454515+0000","last_fullsized":"2023-12-24T21:15:54.454515+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3241,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:15:48.512587+0000","last_change":"2023-12-24T21:15:48.512587+0000","last_active":"2023-12-24T21:15:48.512587+0000","last_peered":"2023-12-24T21:15:48.512587+0000","last_clean":"2023-12-24T21:15:48.512587+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:15:48.512587+0000","last_undegraded":"2023-12-24T21:15:48.512587+0000","last_fullsized":"2023-12-24T21:15:48.512587+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3318,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:52.231439+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T21:12:52.231439+0000","last_peered":"2023-12-24T21:12:52.231439+0000","last_clean":"2023-12-24T21:12:52.231439+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T21:12:52.231439+0000","last_undegraded":"2023-12-24T21:12:52.231439+0000","last_fullsized":"2023-12-24T21:12:52.231439+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740754,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,2,0,4,6,12,6],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:15:19 2023","interfaces":[{"interface":"back","average":{"1min":0.223,"5min":0.221,"15min":0.221},"min":{"1min":0.17100000000000001,"5min":0.154,"15min":0.14399999999999999},"max":{"1min":0.40500000000000003,"5min":0.44500000000000001,"15min":1.198},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.247,"5min":0.247,"15min":0.24399999999999999},"min":{"1min":0.182,"5min":0.161,"15min":0.161},"max":{"1min":0.42599999999999999,"5min":0.45300000000000001,"15min":1.1160000000000001},"last":0.26700000000000002}]},{"osd":1,"last update":"Sun Dec 24 21:15:19 2023","interfaces":[{"interface":"back","average":{"1min":0.26100000000000001,"5min":0.26500000000000001,"15min":0.252},"min":{"1min":0.189,"5min":0.17899999999999999,"15min":0.17699999999999999},"max":{"1min":0.49199999999999999,"5min":1.1499999999999999,"15min":1.1639999999999999},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.28399999999999997,"5min":0.27100000000000002,"15min":0.26700000000000002},"min":{"1min":0.219,"5min":0.187,"15min":0.16900000000000001},"max":{"1min":0.46200000000000002,"5min":0.51400000000000001,"15min":1.153},"last":0.23799999999999999}]}]},{"osd":0,"up_from":8,"seq":34359740753,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:15:22 2023","interfaces":[{"interface":"back","average":{"1min":0.246,"5min":0.23400000000000001,"15min":0.20699999999999999},"min":{"1min":0.158,"5min":0.13900000000000001,"15min":0.13900000000000001},"max":{"1min":0.71599999999999997,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.17899999999999999},{"interface":"front","average":{"1min":0.224,"5min":0.24399999999999999,"15min":0.24199999999999999},"min":{"1min":0.159,"5min":0.159,"15min":0.14799999999999999},"max":{"1min":0.41899999999999998,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.20899999999999999}]},{"osd":2,"last update":"Sun Dec 24 21:15:22 2023","interfaces":[{"interface":"back","average":{"1min":0.246,"5min":0.27700000000000002,"15min":0.26500000000000001},"min":{"1min":0.191,"5min":0.17399999999999999,"15min":0.17100000000000001},"max":{"1min":0.39600000000000002,"5min":1.141,"15min":1.141},"last":0.246},{"interface":"front","average":{"1min":0.27900000000000003,"5min":0.30599999999999999,"15min":0.29499999999999998},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.188},"max":{"1min":0.45200000000000001,"5min":1.157,"15min":1.157},"last":0.27200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359740756,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.193,"5min":0.191,"15min":0.19500000000000001},"min":{"1min":0.151,"5min":0.13300000000000001,"15min":0.126},"max":{"1min":0.249,"5min":0.31900000000000001,"15min":0.437},"last":0.16400000000000001},{"interface":"front","average":{"1min":0.219,"5min":0.221,"15min":0.22500000000000001},"min":{"1min":0.17699999999999999,"5min":0.16900000000000001,"15min":0.153},"max":{"1min":0.27500000000000002,"5min":0.35299999999999998,"15min":0.439},"last":0.19700000000000001}]},{"osd":2,"last update":"Sun Dec 24 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.22,"5min":0.222,"15min":0.23300000000000001},"min":{"1min":0.17699999999999999,"5min":0.16800000000000001,"15min":0.157},"max":{"1min":0.30599999999999999,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.26200000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"min":{"1min":0.23200000000000001,"5min":0.20599999999999999,"15min":0.16900000000000001},"max":{"1min":0.32200000000000001,"5min":0.626,"15min":0.626},"last":0.253}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:16:00.449 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:00.449 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:00.449 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:00.449 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:00.449 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:00.450 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:00.450 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:00.450 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:00.450 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:00.450 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:00.450 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:00.451 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:00.451 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:00.451 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:16:00.980 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:00.982+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:01.133 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:01.134+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:01.405 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:01.406+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:02.012 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:02.013+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:02.097 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:02.099+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:02.397 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:02.399+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:02.992 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:02.994+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:03.054 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:03.055+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:03.424 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:03.425+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:04.006 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:04.006+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:04.037 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:04.038+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:04.398 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:04.398+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:04.995 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:04.995+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:05.025 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:05.025+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:05.411 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:05.412+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:06.037 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:06.038+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:06.074 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:06.075+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:06.427 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:06.428+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:07.070 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:07.070+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:07.091 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:07.092+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:07.450 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:07.451+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:08.048 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:08.048+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:08.093 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:08.093+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:08.451 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:08.452+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:09.067 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:09.067+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:09.090 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:09.090+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:09.416 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:09.417+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:10.065 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:10.066+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:10.086 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:10.086+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:10.371 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:10.372+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:11.085 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:11.085+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:11.105 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:11.105+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:11.333 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:11.334+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:12.042 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:12.042+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:12.134 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:12.135+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:12.309 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:12.309+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:13.071 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:13.071+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:13.109 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:13.109+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:13.312 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:13.312+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:14.062 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:14.062+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:14.074 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:14.074+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:14.281 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:14.282+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:15.046 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:15.046+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:15.100 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:15.100+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:15.249 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:15.250+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:16.053 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:16.053+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:16.064 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:16.065+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:16.268 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:16.268+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:17.066 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:17.066+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:17.077 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:17.077+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:17.291 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:17.291+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:18.030 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:18.030+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:18.067 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:18.068+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:18.254 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:18.255+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:19.013 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:19.013+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:19.097 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:19.098+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:19.222 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:19.222+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:20.042 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:20.042+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:20.089 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:20.089+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:20.173 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:20.174+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:20.451 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:16:20.820 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:16:20.822 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:16:20.834 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":6119,"stamp":"2023-12-24T21:16:20.719119+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,2,0,4,6,12,518],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001837"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2829,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:47.722311+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T21:12:47.722311+0000","last_peered":"2023-12-24T21:12:47.722311+0000","last_clean":"2023-12-24T21:12:47.722311+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T21:12:47.722311+0000","last_undegraded":"2023-12-24T21:12:47.722311+0000","last_fullsized":"2023-12-24T21:12:47.722311+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3623,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:48.083686+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T21:12:48.083686+0000","last_peered":"2023-12-24T21:12:48.083686+0000","last_clean":"2023-12-24T21:12:48.083686+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T21:12:48.083686+0000","last_undegraded":"2023-12-24T21:12:48.083686+0000","last_fullsized":"2023-12-24T21:12:48.083686+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2986,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:48.477240+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T21:12:48.477240+0000","last_peered":"2023-12-24T21:12:48.477240+0000","last_clean":"2023-12-24T21:12:48.477240+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T21:12:48.477240+0000","last_undegraded":"2023-12-24T21:12:48.477240+0000","last_fullsized":"2023-12-24T21:12:48.477240+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10263,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:16:06.429632+0000","last_change":"2023-12-24T21:16:06.429632+0000","last_active":"2023-12-24T21:16:06.429632+0000","last_peered":"2023-12-24T21:16:06.429632+0000","last_clean":"2023-12-24T21:16:06.429632+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:16:06.429632+0000","last_undegraded":"2023-12-24T21:16:06.429632+0000","last_fullsized":"2023-12-24T21:16:06.429632+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3928,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:49.214434+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:12:49.214434+0000","last_peered":"2023-12-24T21:12:49.214434+0000","last_clean":"2023-12-24T21:12:49.214434+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:12:49.214434+0000","last_undegraded":"2023-12-24T21:12:49.214434+0000","last_fullsized":"2023-12-24T21:12:49.214434+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11208,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:49.571330+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T21:12:49.571330+0000","last_peered":"2023-12-24T21:12:49.571330+0000","last_clean":"2023-12-24T21:12:49.571330+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T21:12:49.571330+0000","last_undegraded":"2023-12-24T21:12:49.571330+0000","last_fullsized":"2023-12-24T21:12:49.571330+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":289,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:49.956385+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T21:12:49.956385+0000","last_peered":"2023-12-24T21:12:49.956385+0000","last_clean":"2023-12-24T21:12:49.956385+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T21:12:49.956385+0000","last_undegraded":"2023-12-24T21:12:49.956385+0000","last_fullsized":"2023-12-24T21:12:49.956385+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3534,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:50.314570+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T21:12:50.314570+0000","last_peered":"2023-12-24T21:12:50.314570+0000","last_clean":"2023-12-24T21:12:50.314570+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T21:12:50.314570+0000","last_undegraded":"2023-12-24T21:12:50.314570+0000","last_fullsized":"2023-12-24T21:12:50.314570+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10736,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:50.699675+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T21:12:50.699675+0000","last_peered":"2023-12-24T21:12:50.699675+0000","last_clean":"2023-12-24T21:12:50.699675+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T21:12:50.699675+0000","last_undegraded":"2023-12-24T21:12:50.699675+0000","last_fullsized":"2023-12-24T21:12:50.699675+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3512,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:12:51.052470+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:12:51.052470+0000","last_peered":"2023-12-24T21:12:51.052470+0000","last_clean":"2023-12-24T21:12:51.052470+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:12:51.052470+0000","last_undegraded":"2023-12-24T21:12:51.052470+0000","last_fullsized":"2023-12-24T21:12:51.052470+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 611s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12971,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:16:07.452191+0000","last_change":"2023-12-24T21:16:07.452191+0000","last_active":"2023-12-24T21:16:07.452191+0000","last_peered":"2023-12-24T21:16:07.452191+0000","last_clean":"2023-12-24T21:16:07.452191+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:16:07.452191+0000","last_undegraded":"2023-12-24T21:16:07.452191+0000","last_fullsized":"2023-12-24T21:16:07.452191+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3247,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:16:11.335106+0000","last_change":"2023-12-24T21:16:11.335106+0000","last_active":"2023-12-24T21:16:11.335106+0000","last_peered":"2023-12-24T21:16:11.335106+0000","last_clean":"2023-12-24T21:16:11.335106+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:16:11.335106+0000","last_undegraded":"2023-12-24T21:16:11.335106+0000","last_fullsized":"2023-12-24T21:16:11.335106+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3318,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:52.231439+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T21:12:52.231439+0000","last_peered":"2023-12-24T21:12:52.231439+0000","last_clean":"2023-12-24T21:12:52.231439+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T21:12:52.231439+0000","last_undegraded":"2023-12-24T21:12:52.231439+0000","last_fullsized":"2023-12-24T21:12:52.231439+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740758,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,2,0,4,6,12,6],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:15:19 2023","interfaces":[{"interface":"back","average":{"1min":0.223,"5min":0.221,"15min":0.221},"min":{"1min":0.17100000000000001,"5min":0.154,"15min":0.14399999999999999},"max":{"1min":0.40500000000000003,"5min":0.44500000000000001,"15min":1.198},"last":0.191},{"interface":"front","average":{"1min":0.247,"5min":0.247,"15min":0.24399999999999999},"min":{"1min":0.182,"5min":0.161,"15min":0.161},"max":{"1min":0.42599999999999999,"5min":0.45300000000000001,"15min":1.1160000000000001},"last":0.221}]},{"osd":1,"last update":"Sun Dec 24 21:15:19 2023","interfaces":[{"interface":"back","average":{"1min":0.26100000000000001,"5min":0.26500000000000001,"15min":0.252},"min":{"1min":0.189,"5min":0.17899999999999999,"15min":0.17699999999999999},"max":{"1min":0.49199999999999999,"5min":1.1499999999999999,"15min":1.1639999999999999},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.28399999999999997,"5min":0.27100000000000002,"15min":0.26700000000000002},"min":{"1min":0.219,"5min":0.187,"15min":0.16900000000000001},"max":{"1min":0.46200000000000002,"5min":0.51400000000000001,"15min":1.153},"last":0.25800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359740757,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:15:22 2023","interfaces":[{"interface":"back","average":{"1min":0.246,"5min":0.23400000000000001,"15min":0.20699999999999999},"min":{"1min":0.158,"5min":0.13900000000000001,"15min":0.13900000000000001},"max":{"1min":0.71599999999999997,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.224},{"interface":"front","average":{"1min":0.224,"5min":0.24399999999999999,"15min":0.24199999999999999},"min":{"1min":0.159,"5min":0.159,"15min":0.14799999999999999},"max":{"1min":0.41899999999999998,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.28299999999999997}]},{"osd":2,"last update":"Sun Dec 24 21:15:22 2023","interfaces":[{"interface":"back","average":{"1min":0.246,"5min":0.27700000000000002,"15min":0.26500000000000001},"min":{"1min":0.191,"5min":0.17399999999999999,"15min":0.17100000000000001},"max":{"1min":0.39600000000000002,"5min":1.141,"15min":1.141},"last":0.252},{"interface":"front","average":{"1min":0.27900000000000003,"5min":0.30599999999999999,"15min":0.29499999999999998},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.188},"max":{"1min":0.45200000000000001,"5min":1.157,"15min":1.157},"last":0.315}]}]},{"osd":1,"up_from":8,"seq":34359740760,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.193,"5min":0.191,"15min":0.19500000000000001},"min":{"1min":0.151,"5min":0.13300000000000001,"15min":0.126},"max":{"1min":0.249,"5min":0.31900000000000001,"15min":0.437},"last":0.14099999999999999},{"interface":"front","average":{"1min":0.219,"5min":0.221,"15min":0.22500000000000001},"min":{"1min":0.17699999999999999,"5min":0.16900000000000001,"15min":0.153},"max":{"1min":0.27500000000000002,"5min":0.35299999999999998,"15min":0.439},"last":0.17599999999999999}]},{"osd":2,"last update":"Sun Dec 24 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.22,"5min":0.222,"15min":0.23300000000000001},"min":{"1min":0.17699999999999999,"5min":0.16800000000000001,"15min":0.157},"max":{"1min":0.30599999999999999,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.16300000000000001},{"interface":"front","average":{"1min":0.26200000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"min":{"1min":0.23200000000000001,"5min":0.20599999999999999,"15min":0.16900000000000001},"max":{"1min":0.32200000000000001,"5min":0.626,"15min":0.626},"last":0.21299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:16:20.835 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:20.835 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:20.835 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:20.835 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:20.835 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:20.836 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:20.836 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:20.836 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:20.836 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:20.836 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:20.836 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:20.836 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:20.837 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:20.837 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:16:21.015 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:21.015+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:21.075 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:21.076+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:21.133 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:21.133+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:22.009 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:22.009+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:22.092 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:22.092+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:22.105 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:22.106+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:23.044 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:23.044+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:23.058 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:23.058+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:23.067 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:23.067+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:24.025 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:24.025+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:24.034 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:24.034+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:24.071 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:24.071+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:25.043 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:25.043+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:25.051 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:25.051+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:25.054 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:25.054+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:26.010 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:26.010+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:26.041 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:26.042+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:26.046 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:26.044+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:27.021 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:27.021+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:27.036 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:27.036+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:27.057 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:27.057+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:28.007 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:28.007+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:28.027 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:28.028+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:28.043 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:28.043+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:29.011 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:29.011+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:29.029 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:29.029+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:29.044 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:29.044+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:29.967 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:29.967+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:30.022 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:30.022+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:30.045 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:30.046+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:30.946 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:30.946+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:31.053 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:31.054+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:31.054 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:31.053+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:31.986 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:31.987+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:32.040 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:32.040+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:32.082 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:32.083+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:32.954 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:32.955+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:33.055 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:33.055+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:33.079 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:33.080+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:34.000 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:34.001+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:34.031 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:34.032+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:34.040 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:34.040+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:34.978 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:34.979+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:35.049 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:35.050+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:35.074 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:35.075+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:35.930 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:35.931+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:36.055 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:36.056+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:36.087 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:36.088+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:36.917 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:36.918+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:37.027 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:37.028+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:37.060 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:37.061+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:37.933 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:37.933+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:38.009 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:38.010+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:38.053 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:38.054+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:38.940 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:38.941+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:39.054 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:39.055+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:39.070 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:39.072+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:39.953 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:39.954+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:40.101 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:40.102+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:40.120 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:40.122+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:40.837 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:16:40.969 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:40.970+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:41.079 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:41.080+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:41.144 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:41.145+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:41.158 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:16:41.161 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:16:41.173 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":6129,"stamp":"2023-12-24T21:16:40.722048+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,0,4,6,12,518],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001744"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2829,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:47.722311+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T21:12:47.722311+0000","last_peered":"2023-12-24T21:12:47.722311+0000","last_clean":"2023-12-24T21:12:47.722311+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T21:12:47.722311+0000","last_undegraded":"2023-12-24T21:12:47.722311+0000","last_fullsized":"2023-12-24T21:12:47.722311+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3623,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:48.083686+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T21:12:48.083686+0000","last_peered":"2023-12-24T21:12:48.083686+0000","last_clean":"2023-12-24T21:12:48.083686+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T21:12:48.083686+0000","last_undegraded":"2023-12-24T21:12:48.083686+0000","last_fullsized":"2023-12-24T21:12:48.083686+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2986,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:48.477240+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T21:12:48.477240+0000","last_peered":"2023-12-24T21:12:48.477240+0000","last_clean":"2023-12-24T21:12:48.477240+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T21:12:48.477240+0000","last_undegraded":"2023-12-24T21:12:48.477240+0000","last_fullsized":"2023-12-24T21:12:48.477240+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10272,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:16:37.028065+0000","last_change":"2023-12-24T21:16:37.028065+0000","last_active":"2023-12-24T21:16:37.028065+0000","last_peered":"2023-12-24T21:16:37.028065+0000","last_clean":"2023-12-24T21:16:37.028065+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:16:37.028065+0000","last_undegraded":"2023-12-24T21:16:37.028065+0000","last_fullsized":"2023-12-24T21:16:37.028065+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3928,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:49.214434+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:12:49.214434+0000","last_peered":"2023-12-24T21:12:49.214434+0000","last_clean":"2023-12-24T21:12:49.214434+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:12:49.214434+0000","last_undegraded":"2023-12-24T21:12:49.214434+0000","last_fullsized":"2023-12-24T21:12:49.214434+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11208,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:49.571330+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T21:12:49.571330+0000","last_peered":"2023-12-24T21:12:49.571330+0000","last_clean":"2023-12-24T21:12:49.571330+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T21:12:49.571330+0000","last_undegraded":"2023-12-24T21:12:49.571330+0000","last_fullsized":"2023-12-24T21:12:49.571330+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":289,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:49.956385+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T21:12:49.956385+0000","last_peered":"2023-12-24T21:12:49.956385+0000","last_clean":"2023-12-24T21:12:49.956385+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T21:12:49.956385+0000","last_undegraded":"2023-12-24T21:12:49.956385+0000","last_fullsized":"2023-12-24T21:12:49.956385+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3534,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:50.314570+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T21:12:50.314570+0000","last_peered":"2023-12-24T21:12:50.314570+0000","last_clean":"2023-12-24T21:12:50.314570+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T21:12:50.314570+0000","last_undegraded":"2023-12-24T21:12:50.314570+0000","last_fullsized":"2023-12-24T21:12:50.314570+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10736,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:50.699675+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T21:12:50.699675+0000","last_peered":"2023-12-24T21:12:50.699675+0000","last_clean":"2023-12-24T21:12:50.699675+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T21:12:50.699675+0000","last_undegraded":"2023-12-24T21:12:50.699675+0000","last_fullsized":"2023-12-24T21:12:50.699675+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3512,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:12:51.052470+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:12:51.052470+0000","last_peered":"2023-12-24T21:12:51.052470+0000","last_clean":"2023-12-24T21:12:51.052470+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:12:51.052470+0000","last_undegraded":"2023-12-24T21:12:51.052470+0000","last_fullsized":"2023-12-24T21:12:51.052470+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 611s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12977,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:16:29.045252+0000","last_change":"2023-12-24T21:16:29.045252+0000","last_active":"2023-12-24T21:16:29.045252+0000","last_peered":"2023-12-24T21:16:29.045252+0000","last_clean":"2023-12-24T21:16:29.045252+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:16:29.045252+0000","last_undegraded":"2023-12-24T21:16:29.045252+0000","last_fullsized":"2023-12-24T21:16:29.045252+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3253,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:16:34.032958+0000","last_change":"2023-12-24T21:16:34.032958+0000","last_active":"2023-12-24T21:16:34.032958+0000","last_peered":"2023-12-24T21:16:34.032958+0000","last_clean":"2023-12-24T21:16:34.032958+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:16:34.032958+0000","last_undegraded":"2023-12-24T21:16:34.032958+0000","last_fullsized":"2023-12-24T21:16:34.032958+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3318,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:52.231439+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T21:12:52.231439+0000","last_peered":"2023-12-24T21:12:52.231439+0000","last_clean":"2023-12-24T21:12:52.231439+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T21:12:52.231439+0000","last_undegraded":"2023-12-24T21:12:52.231439+0000","last_fullsized":"2023-12-24T21:12:52.231439+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740762,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,0,4,6,12,6],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:16:20 2023","interfaces":[{"interface":"back","average":{"1min":0.19900000000000001,"5min":0.214,"15min":0.219},"min":{"1min":0.157,"5min":0.154,"15min":0.14399999999999999},"max":{"1min":0.39300000000000002,"5min":0.44500000000000001,"15min":1.198},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.23100000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"min":{"1min":0.18099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":0.33400000000000002,"5min":0.45300000000000001,"15min":1.1160000000000001},"last":0.23100000000000001}]},{"osd":1,"last update":"Sun Dec 24 21:16:20 2023","interfaces":[{"interface":"back","average":{"1min":0.23400000000000001,"5min":0.26100000000000001,"15min":0.251},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.36199999999999999,"5min":1.1499999999999999,"15min":1.1639999999999999},"last":0.20100000000000001},{"interface":"front","average":{"1min":0.247,"5min":0.26500000000000001,"15min":0.26600000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.16900000000000001},"max":{"1min":0.378,"5min":0.51400000000000001,"15min":1.153},"last":0.27000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359740761,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:16:23 2023","interfaces":[{"interface":"back","average":{"1min":0.19900000000000001,"5min":0.23200000000000001,"15min":0.20599999999999999},"min":{"1min":0.161,"5min":0.13900000000000001,"15min":0.13900000000000001},"max":{"1min":0.24399999999999999,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.14000000000000001},{"interface":"front","average":{"1min":0.224,"5min":0.22700000000000001,"15min":0.23999999999999999},"min":{"1min":0.17299999999999999,"5min":0.159,"15min":0.14799999999999999},"max":{"1min":0.28299999999999997,"5min":0.41899999999999998,"15min":1.1120000000000001},"last":0.182}]},{"osd":2,"last update":"Sun Dec 24 21:16:23 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.26300000000000001,"15min":0.26500000000000001},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17100000000000001},"max":{"1min":0.34899999999999998,"5min":1.141,"15min":1.141},"last":0.16400000000000001},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.29099999999999998,"15min":0.29399999999999998},"min":{"1min":0.215,"5min":0.19400000000000001,"15min":0.188},"max":{"1min":0.42099999999999999,"5min":1.157,"15min":1.157},"last":0.20599999999999999}]}]},{"osd":1,"up_from":8,"seq":34359740764,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:16:27 2023","interfaces":[{"interface":"back","average":{"1min":0.184,"5min":0.189,"15min":0.19400000000000001},"min":{"1min":0.14099999999999999,"5min":0.13300000000000001,"15min":0.126},"max":{"1min":0.29299999999999998,"5min":0.31900000000000001,"15min":0.437},"last":0.222},{"interface":"front","average":{"1min":0.222,"5min":0.222,"15min":0.224},"min":{"1min":0.17599999999999999,"5min":0.17399999999999999,"15min":0.153},"max":{"1min":0.33000000000000002,"5min":0.35299999999999998,"15min":0.439},"last":0.247}]},{"osd":2,"last update":"Sun Dec 24 21:16:27 2023","interfaces":[{"interface":"back","average":{"1min":0.22500000000000001,"5min":0.22,"15min":0.23200000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.157},"max":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.63600000000000001},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.25900000000000001,"5min":0.26300000000000001,"15min":0.26700000000000002},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.192},"max":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.626},"last":0.26900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:16:41.174 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:41.174 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:41.174 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:41.175 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:41.175 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:41.175 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:41.175 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:41.175 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:41.175 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:41.176 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:41.176 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:41.176 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:41.176 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:16:41.176 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:16:42.002 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:42.004+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:42.088 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:42.090+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:42.163 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:42.165+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:42.999 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:43.000+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:43.132 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:43.133+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:43.148 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:43.150+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:43.975 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:43.976+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:44.098 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:44.099+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:44.169 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:44.170+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:44.972 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:44.973+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:45.138 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:45.139+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:45.172 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:45.173+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:45.990 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:45.991+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:46.187 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:46.188+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:46.208 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:46.209+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:47.019 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:47.020+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:47.210 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:47.211+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:47.243 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:47.244+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:47.983 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:47.984+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:48.171 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:48.171+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:48.271 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:48.271+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:48.939 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:48.939+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:49.125 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:49.125+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:49.247 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:49.248+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:49.959 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:49.959+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:50.111 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:50.111+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:50.276 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:50.277+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:50.945 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:50.945+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:51.139 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:51.139+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:51.321 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:51.322+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:51.994 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:51.994+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:52.190 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:52.190+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:52.350 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:52.350+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:53.013 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:53.013+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:53.166 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:53.166+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:53.377 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:53.377+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:53.993 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:53.993+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:54.144 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:54.145+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:54.395 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:54.395+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:55.013 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:55.013+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:55.185 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:55.186+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:55.361 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:55.361+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:56.016 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:56.016+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:56.187 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:56.187+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:56.361 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:56.361+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:57.023 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:57.023+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:57.198 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:57.198+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:57.410 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:57.410+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:58.031 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:58.031+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:58.151 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:58.151+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:58.401 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:58.401+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:58.986 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:58.986+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:59.112 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:16:59.112+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:59.427 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:16:59.427+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:16:59.988 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:16:59.988+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:00.118 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:00.119+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:00.418 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:00.418+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:00.964 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:00.965+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:01.141 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:01.141+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:01.177 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:17:01.403 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:01.403+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:01.529 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:17:01.531 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:17:01.542 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":6139,"stamp":"2023-12-24T21:17:00.724873+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,0,4,6,12,518],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001753"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2829,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:47.722311+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T21:12:47.722311+0000","last_peered":"2023-12-24T21:12:47.722311+0000","last_clean":"2023-12-24T21:12:47.722311+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T21:12:47.722311+0000","last_undegraded":"2023-12-24T21:12:47.722311+0000","last_fullsized":"2023-12-24T21:12:47.722311+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3623,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:48.083686+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T21:12:48.083686+0000","last_peered":"2023-12-24T21:12:48.083686+0000","last_clean":"2023-12-24T21:12:48.083686+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T21:12:48.083686+0000","last_undegraded":"2023-12-24T21:12:48.083686+0000","last_fullsized":"2023-12-24T21:12:48.083686+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2986,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:48.477240+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T21:12:48.477240+0000","last_peered":"2023-12-24T21:12:48.477240+0000","last_clean":"2023-12-24T21:12:48.477240+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T21:12:48.477240+0000","last_undegraded":"2023-12-24T21:12:48.477240+0000","last_fullsized":"2023-12-24T21:12:48.477240+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10275,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:16:49.126773+0000","last_change":"2023-12-24T21:16:49.126773+0000","last_active":"2023-12-24T21:16:49.126773+0000","last_peered":"2023-12-24T21:16:49.126773+0000","last_clean":"2023-12-24T21:16:49.126773+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:16:49.126773+0000","last_undegraded":"2023-12-24T21:16:49.126773+0000","last_fullsized":"2023-12-24T21:16:49.126773+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3928,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:49.214434+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:12:49.214434+0000","last_peered":"2023-12-24T21:12:49.214434+0000","last_clean":"2023-12-24T21:12:49.214434+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:12:49.214434+0000","last_undegraded":"2023-12-24T21:12:49.214434+0000","last_fullsized":"2023-12-24T21:12:49.214434+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11208,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:49.571330+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T21:12:49.571330+0000","last_peered":"2023-12-24T21:12:49.571330+0000","last_clean":"2023-12-24T21:12:49.571330+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T21:12:49.571330+0000","last_undegraded":"2023-12-24T21:12:49.571330+0000","last_fullsized":"2023-12-24T21:12:49.571330+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":289,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:49.956385+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T21:12:49.956385+0000","last_peered":"2023-12-24T21:12:49.956385+0000","last_clean":"2023-12-24T21:12:49.956385+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T21:12:49.956385+0000","last_undegraded":"2023-12-24T21:12:49.956385+0000","last_fullsized":"2023-12-24T21:12:49.956385+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3534,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:50.314570+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T21:12:50.314570+0000","last_peered":"2023-12-24T21:12:50.314570+0000","last_clean":"2023-12-24T21:12:50.314570+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T21:12:50.314570+0000","last_undegraded":"2023-12-24T21:12:50.314570+0000","last_fullsized":"2023-12-24T21:12:50.314570+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10736,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:50.699675+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T21:12:50.699675+0000","last_peered":"2023-12-24T21:12:50.699675+0000","last_clean":"2023-12-24T21:12:50.699675+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T21:12:50.699675+0000","last_undegraded":"2023-12-24T21:12:50.699675+0000","last_fullsized":"2023-12-24T21:12:50.699675+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3512,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:12:51.052470+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:12:51.052470+0000","last_peered":"2023-12-24T21:12:51.052470+0000","last_clean":"2023-12-24T21:12:51.052470+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:12:51.052470+0000","last_undegraded":"2023-12-24T21:12:51.052470+0000","last_fullsized":"2023-12-24T21:12:51.052470+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 611s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12983,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:16:50.112168+0000","last_change":"2023-12-24T21:16:50.112168+0000","last_active":"2023-12-24T21:16:50.112168+0000","last_peered":"2023-12-24T21:16:50.112168+0000","last_clean":"2023-12-24T21:16:50.112168+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:16:50.112168+0000","last_undegraded":"2023-12-24T21:16:50.112168+0000","last_fullsized":"2023-12-24T21:16:50.112168+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3259,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:16:52.191305+0000","last_change":"2023-12-24T21:16:52.191305+0000","last_active":"2023-12-24T21:16:52.191305+0000","last_peered":"2023-12-24T21:16:52.191305+0000","last_clean":"2023-12-24T21:16:52.191305+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:16:52.191305+0000","last_undegraded":"2023-12-24T21:16:52.191305+0000","last_fullsized":"2023-12-24T21:16:52.191305+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3318,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:12:52.231439+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T21:12:52.231439+0000","last_peered":"2023-12-24T21:12:52.231439+0000","last_clean":"2023-12-24T21:12:52.231439+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T21:12:52.231439+0000","last_undegraded":"2023-12-24T21:12:52.231439+0000","last_fullsized":"2023-12-24T21:12:52.231439+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740766,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,0,4,6,12,6],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:16:20 2023","interfaces":[{"interface":"back","average":{"1min":0.19900000000000001,"5min":0.214,"15min":0.219},"min":{"1min":0.157,"5min":0.154,"15min":0.14399999999999999},"max":{"1min":0.39300000000000002,"5min":0.44500000000000001,"15min":1.198},"last":0.17199999999999999},{"interface":"front","average":{"1min":0.23100000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"min":{"1min":0.18099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":0.33400000000000002,"5min":0.45300000000000001,"15min":1.1160000000000001},"last":0.193}]},{"osd":1,"last update":"Sun Dec 24 21:16:20 2023","interfaces":[{"interface":"back","average":{"1min":0.23400000000000001,"5min":0.26100000000000001,"15min":0.251},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.36199999999999999,"5min":1.1499999999999999,"15min":1.1639999999999999},"last":0.22},{"interface":"front","average":{"1min":0.247,"5min":0.26500000000000001,"15min":0.26600000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.16900000000000001},"max":{"1min":0.378,"5min":0.51400000000000001,"15min":1.153},"last":0.27900000000000003}]}]},{"osd":0,"up_from":8,"seq":34359740765,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:16:23 2023","interfaces":[{"interface":"back","average":{"1min":0.19900000000000001,"5min":0.23200000000000001,"15min":0.20599999999999999},"min":{"1min":0.161,"5min":0.13900000000000001,"15min":0.13900000000000001},"max":{"1min":0.24399999999999999,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.224,"5min":0.22700000000000001,"15min":0.23999999999999999},"min":{"1min":0.17299999999999999,"5min":0.159,"15min":0.14799999999999999},"max":{"1min":0.28299999999999997,"5min":0.41899999999999998,"15min":1.1120000000000001},"last":0.253}]},{"osd":2,"last update":"Sun Dec 24 21:16:23 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.26300000000000001,"15min":0.26500000000000001},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17100000000000001},"max":{"1min":0.34899999999999998,"5min":1.141,"15min":1.141},"last":0.223},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.29099999999999998,"15min":0.29399999999999998},"min":{"1min":0.215,"5min":0.19400000000000001,"15min":0.188},"max":{"1min":0.42099999999999999,"5min":1.157,"15min":1.157},"last":0.27600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359740768,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:16:27 2023","interfaces":[{"interface":"back","average":{"1min":0.184,"5min":0.189,"15min":0.19400000000000001},"min":{"1min":0.14099999999999999,"5min":0.13300000000000001,"15min":0.126},"max":{"1min":0.29299999999999998,"5min":0.31900000000000001,"15min":0.437},"last":0.216},{"interface":"front","average":{"1min":0.222,"5min":0.222,"15min":0.224},"min":{"1min":0.17599999999999999,"5min":0.17399999999999999,"15min":0.153},"max":{"1min":0.33000000000000002,"5min":0.35299999999999998,"15min":0.439},"last":0.25600000000000001}]},{"osd":2,"last update":"Sun Dec 24 21:16:27 2023","interfaces":[{"interface":"back","average":{"1min":0.22500000000000001,"5min":0.22,"15min":0.23200000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.157},"max":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.63600000000000001},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.25900000000000001,"5min":0.26300000000000001,"15min":0.26700000000000002},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.192},"max":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.626},"last":0.28399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:17:01.544 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:01.544 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:01.544 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:01.544 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:01.545 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:01.545 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:01.545 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:01.545 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:01.546 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:01.546 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:01.546 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:01.546 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:01.546 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:01.547 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.7 2023-12-24T21:17:01.884 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.7 on osd.1 to deep-scrub 2023-12-24T21:17:01.895 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.6 2023-12-24T21:17:01.935 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:01.935+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:02.184 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:02.184+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:02.267 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.6 on osd.1 to deep-scrub 2023-12-24T21:17:02.279 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2023-12-24T21:17:02.364 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:02.364+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:02.699 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2023-12-24T21:17:02.711 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-12-24T21:17:02.942 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:02.942+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:03.063 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-12-24T21:17:03.075 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.0 2023-12-24T21:17:03.186 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:03.186+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:03.347 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:03.347+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:03.421 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.0 on osd.2 to deep-scrub 2023-12-24T21:17:03.432 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-12-24T21:17:03.805 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-12-24T21:17:03.816 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.0 2023-12-24T21:17:03.909 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:03.909+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:04.185 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:04.185+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:04.192 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 1.0 on osd.1 to deep-scrub 2023-12-24T21:17:04.203 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.3 2023-12-24T21:17:04.385 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:04.385+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:04.572 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.3 on osd.1 to deep-scrub 2023-12-24T21:17:04.584 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-12-24T21:17:04.916 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:04.916+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:04.969 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-12-24T21:17:04.981 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.1 2023-12-24T21:17:05.207 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:05.207+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:05.346 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.1 on osd.2 to deep-scrub 2023-12-24T21:17:05.358 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-12-24T21:17:05.420 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:05.420+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:05.739 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-12-24T21:17:05.751 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.2 2023-12-24T21:17:05.945 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:05.945+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:06.102 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.2 on osd.0 to deep-scrub 2023-12-24T21:17:06.109 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.4 2023-12-24T21:17:06.201 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:06.200+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:06.376 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:06.376+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:06.445 INFO:teuthology.orchestra.run.smithi087.stderr:instructing pg 2.4 on osd.1 to deep-scrub 2023-12-24T21:17:06.456 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:17:06.900 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:06.900+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:07.179 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:07.179+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:07.334 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:07.334+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:07.881 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:07.881+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:08.183 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:08.183+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:08.286 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:08.286+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:08.849 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:08.849+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:09.211 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:09.211+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:09.323 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:09.323+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:09.805 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:09.805+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:10.261 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:10.261+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:10.335 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:10.335+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:10.806 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:10.806+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:11.265 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:11.265+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:11.345 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:11.345+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:11.835 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:11.835+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:12.294 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:12.294+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:12.376 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:12.376+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:12.882 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:12.882+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:13.339 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:13.339+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:13.383 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:13.383+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:13.842 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:13.842+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:14.339 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:14.339+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:14.404 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:14.404+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:14.830 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:14.830+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:15.374 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:15.373+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:15.385 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:15.385+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:15.853 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:15.853+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:16.328 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:16.328+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:16.403 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:16.403+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:16.860 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:16.861+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:17.282 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:17.283+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:17.398 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:17.398+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:17.877 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:17.877+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:18.328 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:18.328+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:18.396 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:18.396+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:18.865 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:18.866+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:19.296 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:19.296+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:19.387 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:19.388+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:19.826 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:19.827+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:20.332 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:20.333+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:20.399 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:20.400+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:20.871 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:20.872+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:21.296 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:21.296+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:21.449 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:21.450+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:21.865 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:21.866+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:22.277 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:22.278+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:22.448 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:22.449+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:22.822 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:22.823+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:23.240 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:23.241+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:23.434 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:23.435+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:23.814 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:23.815+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:24.250 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:24.251+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:24.467 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:24.467+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:24.765 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:24.766+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:25.222 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:25.223+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:25.452 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:25.453+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:25.722 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:25.723+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:26.192 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:26.193+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:26.457 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:17:26.494 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:26.495+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:26.724 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:26.725+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:26.804 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:17:26.806 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:17:26.817 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":6152,"stamp":"2023-12-24T21:17:26.728824+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,0,4,6,12,518],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001974"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2830,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:54.796025+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T21:14:54.796025+0000","last_peered":"2023-12-24T21:14:54.796025+0000","last_clean":"2023-12-24T21:14:54.796025+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T21:14:54.796025+0000","last_undegraded":"2023-12-24T21:14:54.796025+0000","last_fullsized":"2023-12-24T21:14:54.796025+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3624,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:55.174874+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T21:14:55.174874+0000","last_peered":"2023-12-24T21:14:55.174874+0000","last_clean":"2023-12-24T21:14:55.174874+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T21:14:55.174874+0000","last_undegraded":"2023-12-24T21:14:55.174874+0000","last_fullsized":"2023-12-24T21:14:55.174874+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2987,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:55.511704+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T21:14:55.511704+0000","last_peered":"2023-12-24T21:14:55.511704+0000","last_clean":"2023-12-24T21:14:55.511704+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T21:14:55.511704+0000","last_undegraded":"2023-12-24T21:14:55.511704+0000","last_fullsized":"2023-12-24T21:14:55.511704+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10285,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:17:16.404436+0000","last_change":"2023-12-24T21:17:16.404436+0000","last_active":"2023-12-24T21:17:16.404436+0000","last_peered":"2023-12-24T21:17:16.404436+0000","last_clean":"2023-12-24T21:17:16.404436+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:17:16.404436+0000","last_undegraded":"2023-12-24T21:17:16.404436+0000","last_fullsized":"2023-12-24T21:17:16.404436+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3929,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:56.295631+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:14:56.295631+0000","last_peered":"2023-12-24T21:14:56.295631+0000","last_clean":"2023-12-24T21:14:56.295631+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:14:56.295631+0000","last_undegraded":"2023-12-24T21:14:56.295631+0000","last_fullsized":"2023-12-24T21:14:56.295631+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11209,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:56.638474+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T21:14:56.638474+0000","last_peered":"2023-12-24T21:14:56.638474+0000","last_clean":"2023-12-24T21:14:56.638474+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T21:14:56.638474+0000","last_undegraded":"2023-12-24T21:14:56.638474+0000","last_fullsized":"2023-12-24T21:14:56.638474+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":290,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:57.036045+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T21:14:57.036045+0000","last_peered":"2023-12-24T21:14:57.036045+0000","last_clean":"2023-12-24T21:14:57.036045+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T21:14:57.036045+0000","last_undegraded":"2023-12-24T21:14:57.036045+0000","last_fullsized":"2023-12-24T21:14:57.036045+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3535,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:57.375609+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T21:14:57.375609+0000","last_peered":"2023-12-24T21:14:57.375609+0000","last_clean":"2023-12-24T21:14:57.375609+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T21:14:57.375609+0000","last_undegraded":"2023-12-24T21:14:57.375609+0000","last_fullsized":"2023-12-24T21:14:57.375609+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10737,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:57.708761+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T21:14:57.708761+0000","last_peered":"2023-12-24T21:14:57.708761+0000","last_clean":"2023-12-24T21:14:57.708761+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T21:14:57.708761+0000","last_undegraded":"2023-12-24T21:14:57.708761+0000","last_fullsized":"2023-12-24T21:14:57.708761+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3513,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:14:58.078520+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:14:58.078520+0000","last_peered":"2023-12-24T21:14:58.078520+0000","last_clean":"2023-12-24T21:14:58.078520+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:14:58.078520+0000","last_undegraded":"2023-12-24T21:14:58.078520+0000","last_fullsized":"2023-12-24T21:14:58.078520+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 739s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12990,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:17:10.262095+0000","last_change":"2023-12-24T21:17:10.262095+0000","last_active":"2023-12-24T21:17:10.262095+0000","last_peered":"2023-12-24T21:17:10.262095+0000","last_clean":"2023-12-24T21:17:10.262095+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:17:10.262095+0000","last_undegraded":"2023-12-24T21:17:10.262095+0000","last_fullsized":"2023-12-24T21:17:10.262095+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3266,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:17:14.340406+0000","last_change":"2023-12-24T21:17:14.340406+0000","last_active":"2023-12-24T21:17:14.340406+0000","last_peered":"2023-12-24T21:17:14.340406+0000","last_clean":"2023-12-24T21:17:14.340406+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:17:14.340406+0000","last_undegraded":"2023-12-24T21:17:14.340406+0000","last_fullsized":"2023-12-24T21:17:14.340406+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3319,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:59.111392+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T21:14:59.111392+0000","last_peered":"2023-12-24T21:14:59.111392+0000","last_clean":"2023-12-24T21:14:59.111392+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T21:14:59.111392+0000","last_undegraded":"2023-12-24T21:14:59.111392+0000","last_fullsized":"2023-12-24T21:14:59.111392+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740771,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,0,4,6,12,6],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:16:20 2023","interfaces":[{"interface":"back","average":{"1min":0.19900000000000001,"5min":0.214,"15min":0.219},"min":{"1min":0.157,"5min":0.154,"15min":0.14399999999999999},"max":{"1min":0.39300000000000002,"5min":0.44500000000000001,"15min":1.198},"last":0.20200000000000001},{"interface":"front","average":{"1min":0.23100000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"min":{"1min":0.18099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":0.33400000000000002,"5min":0.45300000000000001,"15min":1.1160000000000001},"last":0.22}]},{"osd":1,"last update":"Sun Dec 24 21:16:20 2023","interfaces":[{"interface":"back","average":{"1min":0.23400000000000001,"5min":0.26100000000000001,"15min":0.251},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.36199999999999999,"5min":1.1499999999999999,"15min":1.1639999999999999},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.247,"5min":0.26500000000000001,"15min":0.26600000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.16900000000000001},"max":{"1min":0.378,"5min":0.51400000000000001,"15min":1.153},"last":0.25800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359740770,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:16:23 2023","interfaces":[{"interface":"back","average":{"1min":0.19900000000000001,"5min":0.23200000000000001,"15min":0.20599999999999999},"min":{"1min":0.161,"5min":0.13900000000000001,"15min":0.13900000000000001},"max":{"1min":0.24399999999999999,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.182},{"interface":"front","average":{"1min":0.224,"5min":0.22700000000000001,"15min":0.23999999999999999},"min":{"1min":0.17299999999999999,"5min":0.159,"15min":0.14799999999999999},"max":{"1min":0.28299999999999997,"5min":0.41899999999999998,"15min":1.1120000000000001},"last":0.223}]},{"osd":2,"last update":"Sun Dec 24 21:16:23 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.26300000000000001,"15min":0.26500000000000001},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17100000000000001},"max":{"1min":0.34899999999999998,"5min":1.141,"15min":1.141},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.29099999999999998,"15min":0.29399999999999998},"min":{"1min":0.215,"5min":0.19400000000000001,"15min":0.188},"max":{"1min":0.42099999999999999,"5min":1.157,"15min":1.157},"last":0.38}]}]},{"osd":1,"up_from":8,"seq":34359740773,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:16:27 2023","interfaces":[{"interface":"back","average":{"1min":0.184,"5min":0.189,"15min":0.19400000000000001},"min":{"1min":0.14099999999999999,"5min":0.13300000000000001,"15min":0.126},"max":{"1min":0.29299999999999998,"5min":0.31900000000000001,"15min":0.437},"last":0.16700000000000001},{"interface":"front","average":{"1min":0.222,"5min":0.222,"15min":0.224},"min":{"1min":0.17599999999999999,"5min":0.17399999999999999,"15min":0.153},"max":{"1min":0.33000000000000002,"5min":0.35299999999999998,"15min":0.439},"last":0.20899999999999999}]},{"osd":2,"last update":"Sun Dec 24 21:16:27 2023","interfaces":[{"interface":"back","average":{"1min":0.22500000000000001,"5min":0.22,"15min":0.23200000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.157},"max":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.63600000000000001},"last":0.19400000000000001},{"interface":"front","average":{"1min":0.25900000000000001,"5min":0.26300000000000001,"15min":0.26700000000000002},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.192},"max":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.626},"last":0.254}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:17:26.818 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:26.818 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:26.818 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:26.818 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:26.818 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:26.818 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:26.819 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:26.819 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:26.819 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:26.819 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:26.819 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:26.819 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:26.819 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:26.820 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:17:27.210 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:27.211+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:27.529 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:27.531+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:27.747 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:27.748+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:28.242 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:28.243+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:28.500 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:28.501+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:28.782 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:28.784+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:29.256 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:29.257+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:29.469 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:29.470+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:29.785 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:29.786+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:30.256 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:30.257+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:30.478 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:30.479+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:30.830 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:30.831+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:31.250 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:31.250+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:31.489 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:31.491+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:31.879 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:31.880+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:32.300 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:32.301+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:32.534 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:32.535+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:32.863 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:32.863+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:33.347 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:33.347+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:33.520 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:33.520+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:33.871 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:33.871+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:34.330 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:34.330+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:34.480 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:34.480+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:34.899 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:34.899+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:35.284 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:35.284+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:35.436 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:35.436+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:35.897 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:35.897+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:36.273 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:36.273+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:36.395 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:36.395+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:36.849 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:36.849+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:37.253 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:37.253+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:37.430 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:37.430+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:37.857 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:37.857+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:38.303 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:38.303+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:38.398 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:38.398+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:38.810 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:38.809+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:39.342 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:39.342+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:39.423 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:39.423+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:39.854 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:39.854+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:40.384 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:40.383+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:40.420 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:40.420+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:40.896 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:40.896+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:41.358 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:41.358+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:41.423 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:41.422+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:41.889 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:41.888+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:42.372 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:42.371+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:42.453 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:42.452+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:42.928 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:42.928+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:43.382 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:43.381+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:43.463 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:43.463+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:43.895 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:43.895+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:44.358 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:44.358+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:44.454 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:44.454+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:44.900 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:44.899+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:45.366 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:45.365+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:45.472 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:45.471+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:45.943 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:45.942+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 32 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:46.396 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:46.395+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:46.487 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:46.486+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:46.820 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:17:46.951 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:46.951+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:47.139 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:17:47.141 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:17:47.153 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":6162,"stamp":"2023-12-24T21:17:46.731682+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,0,4,6,12,518],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001751"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2830,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:54.796025+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T21:14:54.796025+0000","last_peered":"2023-12-24T21:14:54.796025+0000","last_clean":"2023-12-24T21:14:54.796025+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T21:14:54.796025+0000","last_undegraded":"2023-12-24T21:14:54.796025+0000","last_fullsized":"2023-12-24T21:14:54.796025+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3624,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:55.174874+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T21:14:55.174874+0000","last_peered":"2023-12-24T21:14:55.174874+0000","last_clean":"2023-12-24T21:14:55.174874+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T21:14:55.174874+0000","last_undegraded":"2023-12-24T21:14:55.174874+0000","last_fullsized":"2023-12-24T21:14:55.174874+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2987,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:55.511704+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T21:14:55.511704+0000","last_peered":"2023-12-24T21:14:55.511704+0000","last_clean":"2023-12-24T21:14:55.511704+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T21:14:55.511704+0000","last_undegraded":"2023-12-24T21:14:55.511704+0000","last_fullsized":"2023-12-24T21:14:55.511704+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10291,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:17:40.421383+0000","last_change":"2023-12-24T21:17:40.421383+0000","last_active":"2023-12-24T21:17:40.421383+0000","last_peered":"2023-12-24T21:17:40.421383+0000","last_clean":"2023-12-24T21:17:40.421383+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:17:40.421383+0000","last_undegraded":"2023-12-24T21:17:40.421383+0000","last_fullsized":"2023-12-24T21:17:40.421383+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3929,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:56.295631+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:14:56.295631+0000","last_peered":"2023-12-24T21:14:56.295631+0000","last_clean":"2023-12-24T21:14:56.295631+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:14:56.295631+0000","last_undegraded":"2023-12-24T21:14:56.295631+0000","last_fullsized":"2023-12-24T21:14:56.295631+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11209,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:56.638474+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T21:14:56.638474+0000","last_peered":"2023-12-24T21:14:56.638474+0000","last_clean":"2023-12-24T21:14:56.638474+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T21:14:56.638474+0000","last_undegraded":"2023-12-24T21:14:56.638474+0000","last_fullsized":"2023-12-24T21:14:56.638474+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":290,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:57.036045+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T21:14:57.036045+0000","last_peered":"2023-12-24T21:14:57.036045+0000","last_clean":"2023-12-24T21:14:57.036045+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T21:14:57.036045+0000","last_undegraded":"2023-12-24T21:14:57.036045+0000","last_fullsized":"2023-12-24T21:14:57.036045+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3535,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:57.375609+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T21:14:57.375609+0000","last_peered":"2023-12-24T21:14:57.375609+0000","last_clean":"2023-12-24T21:14:57.375609+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T21:14:57.375609+0000","last_undegraded":"2023-12-24T21:14:57.375609+0000","last_fullsized":"2023-12-24T21:14:57.375609+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10737,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:57.708761+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T21:14:57.708761+0000","last_peered":"2023-12-24T21:14:57.708761+0000","last_clean":"2023-12-24T21:14:57.708761+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T21:14:57.708761+0000","last_undegraded":"2023-12-24T21:14:57.708761+0000","last_fullsized":"2023-12-24T21:14:57.708761+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3513,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:14:58.078520+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:14:58.078520+0000","last_peered":"2023-12-24T21:14:58.078520+0000","last_clean":"2023-12-24T21:14:58.078520+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:14:58.078520+0000","last_undegraded":"2023-12-24T21:14:58.078520+0000","last_fullsized":"2023-12-24T21:14:58.078520+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 739s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":12999,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:17:41.423906+0000","last_change":"2023-12-24T21:17:41.423906+0000","last_active":"2023-12-24T21:17:41.423906+0000","last_peered":"2023-12-24T21:17:41.423906+0000","last_clean":"2023-12-24T21:17:41.423906+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:17:41.423906+0000","last_undegraded":"2023-12-24T21:17:41.423906+0000","last_fullsized":"2023-12-24T21:17:41.423906+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3272,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:17:33.521503+0000","last_change":"2023-12-24T21:17:33.521503+0000","last_active":"2023-12-24T21:17:33.521503+0000","last_peered":"2023-12-24T21:17:33.521503+0000","last_clean":"2023-12-24T21:17:33.521503+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:17:33.521503+0000","last_undegraded":"2023-12-24T21:17:33.521503+0000","last_fullsized":"2023-12-24T21:17:33.521503+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3319,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:59.111392+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T21:14:59.111392+0000","last_peered":"2023-12-24T21:14:59.111392+0000","last_clean":"2023-12-24T21:14:59.111392+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T21:14:59.111392+0000","last_undegraded":"2023-12-24T21:14:59.111392+0000","last_fullsized":"2023-12-24T21:14:59.111392+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740775,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,2,0,4,6,12,6],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:17:22 2023","interfaces":[{"interface":"back","average":{"1min":0.217,"5min":0.21199999999999999,"15min":0.218},"min":{"1min":0.17199999999999999,"5min":0.154,"15min":0.14399999999999999},"max":{"1min":0.30599999999999999,"5min":0.40500000000000003,"15min":1.198},"last":0.311},{"interface":"front","average":{"1min":0.23999999999999999,"5min":0.24099999999999999,"15min":0.24199999999999999},"min":{"1min":0.193,"5min":0.17899999999999999,"15min":0.161},"max":{"1min":0.32300000000000001,"5min":0.42599999999999999,"15min":1.1160000000000001},"last":0.373}]},{"osd":1,"last update":"Sun Dec 24 21:17:22 2023","interfaces":[{"interface":"back","average":{"1min":0.247,"5min":0.25800000000000001,"15min":0.251},"min":{"1min":0.193,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.32100000000000001,"5min":1.1499999999999999,"15min":1.1639999999999999},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.26900000000000002,"5min":0.26300000000000001,"15min":0.26600000000000001},"min":{"1min":0.219,"5min":0.186,"15min":0.16900000000000001},"max":{"1min":0.35599999999999998,"5min":0.46500000000000002,"15min":1.153},"last":0.34200000000000003}]}]},{"osd":0,"up_from":8,"seq":34359740774,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:17:25 2023","interfaces":[{"interface":"back","average":{"1min":0.183,"5min":0.221,"15min":0.20399999999999999},"min":{"1min":0.14000000000000001,"5min":0.14000000000000001,"15min":0.13900000000000001},"max":{"1min":0.308,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.16400000000000001},{"interface":"front","average":{"1min":0.24199999999999999,"5min":0.23000000000000001,"15min":0.24099999999999999},"min":{"1min":0.16700000000000001,"5min":0.159,"15min":0.14799999999999999},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":1.1120000000000001},"last":0.20899999999999999}]},{"osd":2,"last update":"Sun Dec 24 21:17:25 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.26400000000000001,"15min":0.26700000000000002},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":0.66600000000000004,"5min":1.141,"15min":1.141},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.29199999999999998,"15min":0.29599999999999999},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.192},"max":{"1min":0.68300000000000005,"5min":1.157,"15min":1.157},"last":0.26800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359740777,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:17:28 2023","interfaces":[{"interface":"back","average":{"1min":0.20000000000000001,"5min":0.19,"15min":0.193},"min":{"1min":0.14999999999999999,"5min":0.14099999999999999,"15min":0.126},"max":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.437},"last":0.17399999999999999},{"interface":"front","average":{"1min":0.23699999999999999,"5min":0.223,"15min":0.224},"min":{"1min":0.191,"5min":0.17599999999999999,"15min":0.153},"max":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"last":0.21299999999999999}]},{"osd":2,"last update":"Sun Dec 24 21:17:28 2023","interfaces":[{"interface":"back","average":{"1min":0.23200000000000001,"5min":0.221,"15min":0.23000000000000001},"min":{"1min":0.17899999999999999,"5min":0.16300000000000001,"15min":0.157},"max":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.63600000000000001},"last":0.16200000000000001},{"interface":"front","average":{"1min":0.26700000000000002,"5min":0.26200000000000001,"15min":0.26600000000000001},"min":{"1min":0.22800000000000001,"5min":0.20599999999999999,"15min":0.192},"max":{"1min":0.435,"5min":0.435,"15min":0.626},"last":0.23999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:17:47.154 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:47.155 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:47.155 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:47.155 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:47.155 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:47.156 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:47.156 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:47.156 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:47.156 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:47.156 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:47.157 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:47.157 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:47.157 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:17:47.157 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:17:47.440 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:47.440+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:47.444 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:47.444+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:47.928 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:47.928+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:48.442 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:48.442+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:48.447 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:48.447+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:48.881 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:48.881+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:49.438 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:49.438+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:49.476 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:49.476+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:49.882 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:49.882+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:50.458 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:50.458+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:50.527 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:50.526+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:50.892 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:50.892+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:51.495 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:51.495+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:51.542 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:51.542+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:51.859 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:51.859+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:52.519 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:52.519+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:52.521 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:52.521+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:52.892 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:52.892+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:53.480 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:53.480+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:53.500 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:53.500+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:53.862 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:53.862+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:54.459 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:54.458+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:54.504 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:54.504+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:54.846 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:54.846+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:55.497 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:55.496+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:55.509 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:55.508+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:55.841 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:55.841+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:56.481 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:56.480+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:56.517 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:56.517+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:56.862 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:56.862+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:57.504 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:57.504+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:57.510 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:57.510+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:57.891 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:57.891+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:58.476 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:58.475+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:58.519 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:58.519+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:58.906 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:58.906+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:59.457 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:17:59.457+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:59.533 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:17:59.533+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:17:59.935 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:17:59.934+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:00.432 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:00.432+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:00.510 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:00.509+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:00.915 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:00.914+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:01.412 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:01.412+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:01.496 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:01.496+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:01.929 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:01.929+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:02.415 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:02.415+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:02.448 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:02.448+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:02.907 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:02.907+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:03.443 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:03.443+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:03.447 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:03.447+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:03.923 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:03.923+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:04.430 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:04.430+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:04.432 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:04.431+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:04.922 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:04.923+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:05.401 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:05.401+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:05.406 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:05.406+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:05.903 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:05.904+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:06.370 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:06.370+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:06.407 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:06.408+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:06.891 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:06.889+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:07.158 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:18:07.387 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:07.387+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:07.412 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:07.413+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:07.492 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:18:07.494 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:18:07.503 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":6172,"stamp":"2023-12-24T21:18:06.734542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,2,2,0,4,6,12,518],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001777"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2830,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:54.796025+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T21:14:54.796025+0000","last_peered":"2023-12-24T21:14:54.796025+0000","last_clean":"2023-12-24T21:14:54.796025+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T21:14:54.796025+0000","last_undegraded":"2023-12-24T21:14:54.796025+0000","last_fullsized":"2023-12-24T21:14:54.796025+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3624,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:55.174874+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T21:14:55.174874+0000","last_peered":"2023-12-24T21:14:55.174874+0000","last_clean":"2023-12-24T21:14:55.174874+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T21:14:55.174874+0000","last_undegraded":"2023-12-24T21:14:55.174874+0000","last_fullsized":"2023-12-24T21:14:55.174874+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2987,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:55.511704+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T21:14:55.511704+0000","last_peered":"2023-12-24T21:14:55.511704+0000","last_clean":"2023-12-24T21:14:55.511704+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T21:14:55.511704+0000","last_undegraded":"2023-12-24T21:14:55.511704+0000","last_fullsized":"2023-12-24T21:14:55.511704+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10297,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:17:57.504898+0000","last_change":"2023-12-24T21:17:57.504898+0000","last_active":"2023-12-24T21:17:57.504898+0000","last_peered":"2023-12-24T21:17:57.504898+0000","last_clean":"2023-12-24T21:17:57.504898+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:17:57.504898+0000","last_undegraded":"2023-12-24T21:17:57.504898+0000","last_fullsized":"2023-12-24T21:17:57.504898+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3929,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:56.295631+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:14:56.295631+0000","last_peered":"2023-12-24T21:14:56.295631+0000","last_clean":"2023-12-24T21:14:56.295631+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:14:56.295631+0000","last_undegraded":"2023-12-24T21:14:56.295631+0000","last_fullsized":"2023-12-24T21:14:56.295631+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11209,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:56.638474+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T21:14:56.638474+0000","last_peered":"2023-12-24T21:14:56.638474+0000","last_clean":"2023-12-24T21:14:56.638474+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T21:14:56.638474+0000","last_undegraded":"2023-12-24T21:14:56.638474+0000","last_fullsized":"2023-12-24T21:14:56.638474+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":290,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:57.036045+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T21:14:57.036045+0000","last_peered":"2023-12-24T21:14:57.036045+0000","last_clean":"2023-12-24T21:14:57.036045+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T21:14:57.036045+0000","last_undegraded":"2023-12-24T21:14:57.036045+0000","last_fullsized":"2023-12-24T21:14:57.036045+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3535,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:57.375609+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T21:14:57.375609+0000","last_peered":"2023-12-24T21:14:57.375609+0000","last_clean":"2023-12-24T21:14:57.375609+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T21:14:57.375609+0000","last_undegraded":"2023-12-24T21:14:57.375609+0000","last_fullsized":"2023-12-24T21:14:57.375609+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10737,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:57.708761+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T21:14:57.708761+0000","last_peered":"2023-12-24T21:14:57.708761+0000","last_clean":"2023-12-24T21:14:57.708761+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T21:14:57.708761+0000","last_undegraded":"2023-12-24T21:14:57.708761+0000","last_fullsized":"2023-12-24T21:14:57.708761+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3513,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:14:58.078520+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:14:58.078520+0000","last_peered":"2023-12-24T21:14:58.078520+0000","last_clean":"2023-12-24T21:14:58.078520+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:14:58.078520+0000","last_undegraded":"2023-12-24T21:14:58.078520+0000","last_fullsized":"2023-12-24T21:14:58.078520+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 739s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":13005,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:17:58.520565+0000","last_change":"2023-12-24T21:17:58.520565+0000","last_active":"2023-12-24T21:17:58.520565+0000","last_peered":"2023-12-24T21:17:58.520565+0000","last_clean":"2023-12-24T21:17:58.520565+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:17:58.520565+0000","last_undegraded":"2023-12-24T21:17:58.520565+0000","last_fullsized":"2023-12-24T21:17:58.520565+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3281,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:18:00.510405+0000","last_change":"2023-12-24T21:18:00.510405+0000","last_active":"2023-12-24T21:18:00.510405+0000","last_peered":"2023-12-24T21:18:00.510405+0000","last_clean":"2023-12-24T21:18:00.510405+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:18:00.510405+0000","last_undegraded":"2023-12-24T21:18:00.510405+0000","last_fullsized":"2023-12-24T21:18:00.510405+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3319,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:59.111392+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T21:14:59.111392+0000","last_peered":"2023-12-24T21:14:59.111392+0000","last_clean":"2023-12-24T21:14:59.111392+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T21:14:59.111392+0000","last_undegraded":"2023-12-24T21:14:59.111392+0000","last_fullsized":"2023-12-24T21:14:59.111392+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740779,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,2,2,0,4,6,12,6],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:17:22 2023","interfaces":[{"interface":"back","average":{"1min":0.217,"5min":0.21199999999999999,"15min":0.218},"min":{"1min":0.17199999999999999,"5min":0.154,"15min":0.14399999999999999},"max":{"1min":0.30599999999999999,"5min":0.40500000000000003,"15min":1.198},"last":0.17699999999999999},{"interface":"front","average":{"1min":0.23999999999999999,"5min":0.24099999999999999,"15min":0.24199999999999999},"min":{"1min":0.193,"5min":0.17899999999999999,"15min":0.161},"max":{"1min":0.32300000000000001,"5min":0.42599999999999999,"15min":1.1160000000000001},"last":0.23599999999999999}]},{"osd":1,"last update":"Sun Dec 24 21:17:22 2023","interfaces":[{"interface":"back","average":{"1min":0.247,"5min":0.25800000000000001,"15min":0.251},"min":{"1min":0.193,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.32100000000000001,"5min":1.1499999999999999,"15min":1.1639999999999999},"last":0.217},{"interface":"front","average":{"1min":0.26900000000000002,"5min":0.26300000000000001,"15min":0.26600000000000001},"min":{"1min":0.219,"5min":0.186,"15min":0.16900000000000001},"max":{"1min":0.35599999999999998,"5min":0.46500000000000002,"15min":1.153},"last":0.27400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359740778,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:17:25 2023","interfaces":[{"interface":"back","average":{"1min":0.183,"5min":0.221,"15min":0.20399999999999999},"min":{"1min":0.14000000000000001,"5min":0.14000000000000001,"15min":0.13900000000000001},"max":{"1min":0.308,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.17299999999999999},{"interface":"front","average":{"1min":0.24199999999999999,"5min":0.23000000000000001,"15min":0.24099999999999999},"min":{"1min":0.16700000000000001,"5min":0.159,"15min":0.14799999999999999},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":1.1120000000000001},"last":0.217}]},{"osd":2,"last update":"Sun Dec 24 21:17:25 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.26400000000000001,"15min":0.26700000000000002},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":0.66600000000000004,"5min":1.141,"15min":1.141},"last":0.248},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.29199999999999998,"15min":0.29599999999999999},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.192},"max":{"1min":0.68300000000000005,"5min":1.157,"15min":1.157},"last":0.26600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359740781,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:17:28 2023","interfaces":[{"interface":"back","average":{"1min":0.20000000000000001,"5min":0.19,"15min":0.193},"min":{"1min":0.14999999999999999,"5min":0.14099999999999999,"15min":0.126},"max":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.437},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.23699999999999999,"5min":0.223,"15min":0.224},"min":{"1min":0.191,"5min":0.17599999999999999,"15min":0.153},"max":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"last":0.20799999999999999}]},{"osd":2,"last update":"Sun Dec 24 21:17:28 2023","interfaces":[{"interface":"back","average":{"1min":0.23200000000000001,"5min":0.221,"15min":0.23000000000000001},"min":{"1min":0.17899999999999999,"5min":0.16300000000000001,"15min":0.157},"max":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.63600000000000001},"last":0.185},{"interface":"front","average":{"1min":0.26700000000000002,"5min":0.26200000000000001,"15min":0.26600000000000001},"min":{"1min":0.22800000000000001,"5min":0.20599999999999999,"15min":0.192},"max":{"1min":0.435,"5min":0.435,"15min":0.626},"last":0.23999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:18:07.505 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:07.505 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:07.505 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:07.506 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:07.506 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:07.506 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:07.506 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:07.506 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:07.506 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:07.506 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:07.506 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:07.507 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:07.507 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:07.507 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:18:07.928 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:07.928+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:08.356 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:08.356+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:08.408 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:08.409+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:08.953 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:08.952+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:09.382 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:09.383+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:09.428 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:09.429+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:09.933 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:09.933+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:10.346 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:10.345+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:10.467 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:10.467+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:10.919 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:10.919+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:11.360 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:11.361+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:11.453 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:11.453+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:11.953 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:11.954+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:12.324 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:12.324+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:12.427 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:12.427+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:12.952 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:12.953+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:13.280 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:13.281+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:13.444 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:13.444+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:13.971 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:13.972+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:14.250 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:14.251+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:14.408 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:14.408+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:14.928 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:14.928+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:15.230 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:15.231+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:15.374 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:15.375+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:15.931 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:15.931+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 15 slow ops, oldest is osd_op(osd.0.8:15047 2.1 2:9da1c2e7:::rbd_data.2973596cc8be.000000000000005d:head [copy-from ver 4296 in=76b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:16.261 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:16.261+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:16.406 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:16.407+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:16.972 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:16.972+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 34 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:17.279 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:17.280+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:17.439 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:17.440+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:18.002 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:18.002+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 34 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:18.241 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:18.241+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:18.424 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:18.424+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:18.976 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:18.976+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 34 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:19.269 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:19.268+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:19.438 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:19.437+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:19.943 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:19.943+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 34 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:20.222 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:20.222+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:20.468 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:20.468+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:20.943 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:20.943+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 34 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:21.220 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:21.219+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:21.485 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:21.485+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:21.913 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:21.913+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 34 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:22.189 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:22.189+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:22.468 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:22.468+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:22.905 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:22.905+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 34 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:23.203 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:23.202+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:23.482 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:23.482+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:23.887 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:23.887+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 34 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:24.232 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:24.231+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:24.506 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:24.505+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:24.872 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:24.871+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 34 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:25.266 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:25.265+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:25.528 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:25.527+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:25.833 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:25.833+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 34 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:26.307 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:26.306+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:26.532 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:26.532+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:26.850 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:26.850+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 34 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:27.332 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:27.331+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:27.508 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:18:27.535 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:27.534+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:27.855 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:27.855+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 34 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:27.872 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:18:27.875 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:18:27.886 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":6182,"stamp":"2023-12-24T21:18:26.737575+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,2,2,0,4,6,12,518],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001947"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2830,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:54.796025+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T21:14:54.796025+0000","last_peered":"2023-12-24T21:14:54.796025+0000","last_clean":"2023-12-24T21:14:54.796025+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T21:14:54.796025+0000","last_undegraded":"2023-12-24T21:14:54.796025+0000","last_fullsized":"2023-12-24T21:14:54.796025+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3624,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:55.174874+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T21:14:55.174874+0000","last_peered":"2023-12-24T21:14:55.174874+0000","last_clean":"2023-12-24T21:14:55.174874+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T21:14:55.174874+0000","last_undegraded":"2023-12-24T21:14:55.174874+0000","last_fullsized":"2023-12-24T21:14:55.174874+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2987,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:55.511704+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T21:14:55.511704+0000","last_peered":"2023-12-24T21:14:55.511704+0000","last_clean":"2023-12-24T21:14:55.511704+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T21:14:55.511704+0000","last_undegraded":"2023-12-24T21:14:55.511704+0000","last_fullsized":"2023-12-24T21:14:55.511704+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10300,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:18:02.449543+0000","last_change":"2023-12-24T21:18:02.449543+0000","last_active":"2023-12-24T21:18:02.449543+0000","last_peered":"2023-12-24T21:18:02.449543+0000","last_clean":"2023-12-24T21:18:02.449543+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:18:02.449543+0000","last_undegraded":"2023-12-24T21:18:02.449543+0000","last_fullsized":"2023-12-24T21:18:02.449543+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3929,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:56.295631+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:14:56.295631+0000","last_peered":"2023-12-24T21:14:56.295631+0000","last_clean":"2023-12-24T21:14:56.295631+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:14:56.295631+0000","last_undegraded":"2023-12-24T21:14:56.295631+0000","last_fullsized":"2023-12-24T21:14:56.295631+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11209,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:56.638474+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T21:14:56.638474+0000","last_peered":"2023-12-24T21:14:56.638474+0000","last_clean":"2023-12-24T21:14:56.638474+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T21:14:56.638474+0000","last_undegraded":"2023-12-24T21:14:56.638474+0000","last_fullsized":"2023-12-24T21:14:56.638474+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":290,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:57.036045+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T21:14:57.036045+0000","last_peered":"2023-12-24T21:14:57.036045+0000","last_clean":"2023-12-24T21:14:57.036045+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T21:14:57.036045+0000","last_undegraded":"2023-12-24T21:14:57.036045+0000","last_fullsized":"2023-12-24T21:14:57.036045+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3535,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:57.375609+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T21:14:57.375609+0000","last_peered":"2023-12-24T21:14:57.375609+0000","last_clean":"2023-12-24T21:14:57.375609+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T21:14:57.375609+0000","last_undegraded":"2023-12-24T21:14:57.375609+0000","last_fullsized":"2023-12-24T21:14:57.375609+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10737,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:57.708761+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T21:14:57.708761+0000","last_peered":"2023-12-24T21:14:57.708761+0000","last_clean":"2023-12-24T21:14:57.708761+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T21:14:57.708761+0000","last_undegraded":"2023-12-24T21:14:57.708761+0000","last_fullsized":"2023-12-24T21:14:57.708761+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3513,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:14:58.078520+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:14:58.078520+0000","last_peered":"2023-12-24T21:14:58.078520+0000","last_clean":"2023-12-24T21:14:58.078520+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:14:58.078520+0000","last_undegraded":"2023-12-24T21:14:58.078520+0000","last_fullsized":"2023-12-24T21:14:58.078520+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 739s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":13008,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:18:10.467790+0000","last_change":"2023-12-24T21:18:10.467790+0000","last_active":"2023-12-24T21:18:10.467790+0000","last_peered":"2023-12-24T21:18:10.467790+0000","last_clean":"2023-12-24T21:18:10.467790+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:18:10.467790+0000","last_undegraded":"2023-12-24T21:18:10.467790+0000","last_fullsized":"2023-12-24T21:18:10.467790+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3284,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:18:11.453221+0000","last_change":"2023-12-24T21:18:11.453221+0000","last_active":"2023-12-24T21:18:11.453221+0000","last_peered":"2023-12-24T21:18:11.453221+0000","last_clean":"2023-12-24T21:18:11.453221+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:18:11.453221+0000","last_undegraded":"2023-12-24T21:18:11.453221+0000","last_fullsized":"2023-12-24T21:18:11.453221+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3319,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:59.111392+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T21:14:59.111392+0000","last_peered":"2023-12-24T21:14:59.111392+0000","last_clean":"2023-12-24T21:14:59.111392+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T21:14:59.111392+0000","last_undegraded":"2023-12-24T21:14:59.111392+0000","last_fullsized":"2023-12-24T21:14:59.111392+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740783,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,2,2,0,4,6,12,6],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:17:22 2023","interfaces":[{"interface":"back","average":{"1min":0.217,"5min":0.21199999999999999,"15min":0.218},"min":{"1min":0.17199999999999999,"5min":0.154,"15min":0.14399999999999999},"max":{"1min":0.30599999999999999,"5min":0.40500000000000003,"15min":1.198},"last":0.307},{"interface":"front","average":{"1min":0.23999999999999999,"5min":0.24099999999999999,"15min":0.24199999999999999},"min":{"1min":0.193,"5min":0.17899999999999999,"15min":0.161},"max":{"1min":0.32300000000000001,"5min":0.42599999999999999,"15min":1.1160000000000001},"last":0.38600000000000001}]},{"osd":1,"last update":"Sun Dec 24 21:17:22 2023","interfaces":[{"interface":"back","average":{"1min":0.247,"5min":0.25800000000000001,"15min":0.251},"min":{"1min":0.193,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.32100000000000001,"5min":1.1499999999999999,"15min":1.1639999999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.26900000000000002,"5min":0.26300000000000001,"15min":0.26600000000000001},"min":{"1min":0.219,"5min":0.186,"15min":0.16900000000000001},"max":{"1min":0.35599999999999998,"5min":0.46500000000000002,"15min":1.153},"last":0.34899999999999998}]}]},{"osd":0,"up_from":8,"seq":34359740782,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:17:25 2023","interfaces":[{"interface":"back","average":{"1min":0.183,"5min":0.221,"15min":0.20399999999999999},"min":{"1min":0.14000000000000001,"5min":0.14000000000000001,"15min":0.13900000000000001},"max":{"1min":0.308,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.19400000000000001},{"interface":"front","average":{"1min":0.24199999999999999,"5min":0.23000000000000001,"15min":0.24099999999999999},"min":{"1min":0.16700000000000001,"5min":0.159,"15min":0.14799999999999999},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":1.1120000000000001},"last":0.24399999999999999}]},{"osd":2,"last update":"Sun Dec 24 21:17:25 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.26400000000000001,"15min":0.26700000000000002},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":0.66600000000000004,"5min":1.141,"15min":1.141},"last":0.218},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.29199999999999998,"15min":0.29599999999999999},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.192},"max":{"1min":0.68300000000000005,"5min":1.157,"15min":1.157},"last":0.26800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359740785,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:17:28 2023","interfaces":[{"interface":"back","average":{"1min":0.20000000000000001,"5min":0.19,"15min":0.193},"min":{"1min":0.14999999999999999,"5min":0.14099999999999999,"15min":0.126},"max":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.437},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.23699999999999999,"5min":0.223,"15min":0.224},"min":{"1min":0.191,"5min":0.17599999999999999,"15min":0.153},"max":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"last":0.42399999999999999}]},{"osd":2,"last update":"Sun Dec 24 21:17:28 2023","interfaces":[{"interface":"back","average":{"1min":0.23200000000000001,"5min":0.221,"15min":0.23000000000000001},"min":{"1min":0.17899999999999999,"5min":0.16300000000000001,"15min":0.157},"max":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.63600000000000001},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.26700000000000002,"5min":0.26200000000000001,"15min":0.26600000000000001},"min":{"1min":0.22800000000000001,"5min":0.20599999999999999,"15min":0.192},"max":{"1min":0.435,"5min":0.435,"15min":0.626},"last":0.48799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:18:27.888 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:27.888 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:27.888 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:27.888 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:27.889 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:27.889 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:27.889 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:27.889 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:27.890 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:27.890 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:27.890 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:27.890 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:27.890 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:27.891 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-24T21:18:28.304 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:28.303+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:28.492 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:28.491+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:28.828 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:28.828+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 34 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:29.302 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:29.302+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:29.476 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:29.476+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:29.847 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:29.846+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 34 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:30.324 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:30.323+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:30.470 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:30.470+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:30.865 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:30.864+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 34 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:31.307 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:31.305+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:31.439 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:31.438+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:31.823 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:31.822+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 34 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:32.339 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:32.338+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:32.489 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:32.489+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:32.792 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:32.792+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 34 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:33.358 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:33.357+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:33.479 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:33.479+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:33.817 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:33.816+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 34 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:34.319 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:34.318+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:34.504 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:34.504+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:34.813 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:34.813+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 34 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:35.327 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:35.327+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:35.466 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:35.466+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:35.814 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:35.813+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 34 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:36.300 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:36.299+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:36.443 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:36.443+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:36.797 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:36.796+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 34 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:37.291 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:37.291+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:37.420 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:37.420+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:37.806 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:37.806+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 34 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:38.302 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:38.302+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:38.407 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:38.407+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:38.782 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:38.782+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 34 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:39.266 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:39.266+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:39.445 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:39.444+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:39.752 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:39.752+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 34 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:40.298 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:40.298+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:40.476 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:40.476+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:40.785 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:40.784+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 34 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:41.343 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:41.342+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:41.483 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:41.482+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:41.748 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:41.747+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 34 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:42.365 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:42.364+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:42.453 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:42.452+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:42.774 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:42.773+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 34 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:43.393 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:43.392+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:43.454 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:43.454+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:43.733 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:43.732+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 34 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:44.403 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:44.402+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:44.407 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:44.406+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:44.738 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:44.737+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 34 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:45.429 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:45.428+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:45.454 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:45.454+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:45.756 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:45.756+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 34 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:46.396 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:46.395+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:46.489 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:46.489+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:46.771 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:46.771+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 34 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:47.354 INFO:tasks.ceph.osd.1.smithi087.stderr:2023-12-24T21:18:47.353+0000 7f7f6d9cc700 -1 osd.1 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:3386 3.2 3:518bf5d1:::rbd_data.2973596cc8be.0000000000000071:head [set-alloc-hint object_size 4194304 write_size 4194304,write 708608~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:47.462 INFO:tasks.ceph.osd.0.smithi087.stderr:2023-12-24T21:18:47.462+0000 7f6da0215700 -1 osd.0 162 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.10614.0:4088 3.1 3:afc7fed5:::rbd_data.2973596cc8be.0000000000000060:head [write 765952~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:47.725 INFO:tasks.ceph.osd.2.smithi087.stderr:2023-12-24T21:18:47.724+0000 7f621ffaa700 -1 osd.2 162 get_health_metrics reporting 34 slow ops, oldest is osd_op(osd.0.8:15043 2.1 2:801df84a:test1::rbd_data.295c9a9dfb18.00000000000000e3:head [copy-from ver 3824 in=81b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e159) 2023-12-24T21:18:47.891 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-24T21:18:48.222 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-24T21:18:48.224 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-24T21:18:48.236 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":6192,"stamp":"2023-12-24T21:18:46.740934+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2233312801,"num_objects":789,"num_object_clones":1,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":150,"num_read":68322,"num_read_kb":453930,"num_write":30209,"num_write_kb":8951382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37389,"ondisk_log_size":37389,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586036,"kb_used_data":665500,"kb_used_omap":0,"kb_used_meta":920512,"kb_avail":281529484,"statfs":{"total":289910292480,"available":288286191616,"internally_reserved":0,"allocated":681472000,"data_stored":679117135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":942604288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,4,0,4,6,12,518],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001889"},"pg_stats":[{"pgid":"2.7","version":"159'1813","reported_seq":2830,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:54.796025+0000","last_change":"2023-12-24T18:02:38.125901+0000","last_active":"2023-12-24T21:14:54.796025+0000","last_peered":"2023-12-24T21:14:54.796025+0000","last_clean":"2023-12-24T21:14:54.796025+0000","last_became_active":"2023-12-24T17:57:44.101149+0000","last_became_peered":"2023-12-24T17:57:44.101149+0000","last_unstale":"2023-12-24T21:14:54.796025+0000","last_undegraded":"2023-12-24T21:14:54.796025+0000","last_fullsized":"2023-12-24T21:14:54.796025+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":1813,"log_dups_size":0,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048327599999999999,"stat_sum":{"num_bytes":183152675,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":791,"num_read_kb":13090,"num_write":1684,"num_write_kb":1194695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'2687","reported_seq":3624,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:55.174874+0000","last_change":"2023-12-24T18:02:38.125767+0000","last_active":"2023-12-24T21:14:55.174874+0000","last_peered":"2023-12-24T21:14:55.174874+0000","last_clean":"2023-12-24T21:14:55.174874+0000","last_became_active":"2023-12-24T17:57:44.100979+0000","last_became_peered":"2023-12-24T17:57:44.100979+0000","last_unstale":"2023-12-24T21:14:55.174874+0000","last_undegraded":"2023-12-24T21:14:55.174874+0000","last_fullsized":"2023-12-24T21:14:55.174874+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2687,"log_dups_size":0,"ondisk_log_size":2687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000767992,"stat_sum":{"num_bytes":310758257,"num_objects":86,"num_object_clones":1,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":1,"num_read":648,"num_read_kb":12125,"num_write":2504,"num_write_kb":1031966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'2003","reported_seq":2987,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:55.511704+0000","last_change":"2023-12-24T18:02:38.125270+0000","last_active":"2023-12-24T21:14:55.511704+0000","last_peered":"2023-12-24T21:14:55.511704+0000","last_clean":"2023-12-24T21:14:55.511704+0000","last_became_active":"2023-12-24T17:57:44.101020+0000","last_became_peered":"2023-12-24T17:57:44.101020+0000","last_unstale":"2023-12-24T21:14:55.511704+0000","last_undegraded":"2023-12-24T21:14:55.511704+0000","last_fullsized":"2023-12-24T21:14:55.511704+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031709799999999998,"stat_sum":{"num_bytes":222044380,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":740,"num_read_kb":7097,"num_write":1878,"num_write_kb":1066343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"159'4015","reported_seq":10306,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:18:33.480363+0000","last_change":"2023-12-24T21:18:33.480363+0000","last_active":"2023-12-24T21:18:33.480363+0000","last_peered":"2023-12-24T21:18:33.480363+0000","last_clean":"2023-12-24T21:18:33.480363+0000","last_became_active":"2023-12-24T17:57:51.124872+0000","last_became_peered":"2023-12-24T17:57:51.124872+0000","last_unstale":"2023-12-24T21:18:33.480363+0000","last_undegraded":"2023-12-24T21:18:33.480363+0000","last_fullsized":"2023-12-24T21:18:33.480363+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055541,"stat_sum":{"num_bytes":60736538,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":37,"num_read":14980,"num_read_kb":69219,"num_write":2780,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":952,"num_flush":210,"num_flush_kb":14108,"num_evict":549,"num_evict_kb":37420,"num_promote":603,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"159'2835","reported_seq":3929,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:56.295631+0000","last_change":"2023-12-24T18:02:38.129253+0000","last_active":"2023-12-24T21:14:56.295631+0000","last_peered":"2023-12-24T21:14:56.295631+0000","last_clean":"2023-12-24T21:14:56.295631+0000","last_became_active":"2023-12-24T17:57:44.102578+0000","last_became_peered":"2023-12-24T17:57:44.102578+0000","last_unstale":"2023-12-24T21:14:56.295631+0000","last_undegraded":"2023-12-24T21:14:56.295631+0000","last_fullsized":"2023-12-24T21:14:56.295631+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030009299999999997,"stat_sum":{"num_bytes":280989902,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":915,"num_read_kb":10961,"num_write":2274,"num_write_kb":1274780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"160'4052","reported_seq":11209,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:56.638474+0000","last_change":"2023-12-24T18:02:38.126328+0000","last_active":"2023-12-24T21:14:56.638474+0000","last_peered":"2023-12-24T21:14:56.638474+0000","last_clean":"2023-12-24T21:14:56.638474+0000","last_became_active":"2023-12-24T17:57:51.124870+0000","last_became_peered":"2023-12-24T17:57:51.124870+0000","last_unstale":"2023-12-24T21:14:56.638474+0000","last_undegraded":"2023-12-24T21:14:56.638474+0000","last_fullsized":"2023-12-24T21:14:56.638474+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032575699999999998,"stat_sum":{"num_bytes":31909200,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":37,"num_read":14202,"num_read_kb":120263,"num_write":2106,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":1360,"num_flush":188,"num_flush_kb":41882,"num_evict":512,"num_evict_kb":48868,"num_promote":558,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":290,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:57.036045+0000","last_change":"2023-12-24T17:57:39.279966+0000","last_active":"2023-12-24T21:14:57.036045+0000","last_peered":"2023-12-24T21:14:57.036045+0000","last_clean":"2023-12-24T21:14:57.036045+0000","last_became_active":"2023-12-24T17:57:39.278786+0000","last_became_peered":"2023-12-24T17:57:39.278786+0000","last_unstale":"2023-12-24T21:14:57.036045+0000","last_undegraded":"2023-12-24T21:14:57.036045+0000","last_fullsized":"2023-12-24T21:14:57.036045+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:37.923951+0000","last_clean_scrub_stamp":"2023-12-24T17:57:37.923951+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'2568","reported_seq":3535,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:57.375609+0000","last_change":"2023-12-24T18:02:38.125518+0000","last_active":"2023-12-24T21:14:57.375609+0000","last_peered":"2023-12-24T21:14:57.375609+0000","last_clean":"2023-12-24T21:14:57.375609+0000","last_became_active":"2023-12-24T17:57:44.100677+0000","last_became_peered":"2023-12-24T17:57:44.100677+0000","last_unstale":"2023-12-24T21:14:57.375609+0000","last_undegraded":"2023-12-24T21:14:57.375609+0000","last_fullsized":"2023-12-24T21:14:57.375609+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053638399999999999,"stat_sum":{"num_bytes":261148688,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":736,"num_read_kb":6834,"num_write":2024,"num_write_kb":1087465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'5148","reported_seq":10737,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:57.708761+0000","last_change":"2023-12-24T18:02:38.126033+0000","last_active":"2023-12-24T21:14:57.708761+0000","last_peered":"2023-12-24T21:14:57.708761+0000","last_clean":"2023-12-24T21:14:57.708761+0000","last_became_active":"2023-12-24T17:57:51.124972+0000","last_became_peered":"2023-12-24T17:57:51.124972+0000","last_unstale":"2023-12-24T21:14:57.708761+0000","last_undegraded":"2023-12-24T21:14:57.708761+0000","last_fullsized":"2023-12-24T21:14:57.708761+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060284399999999995,"stat_sum":{"num_bytes":32478927,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":40,"num_read":12787,"num_read_kb":77305,"num_write":3436,"num_write_kb":2492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":1645,"num_flush":197,"num_flush_kb":13727,"num_evict":475,"num_evict_kb":76816,"num_promote":523,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"159'2468","reported_seq":3513,"reported_epoch":162,"state":"active+clean+scrubbing+deep","last_fresh":"2023-12-24T21:14:58.078520+0000","last_change":"2023-12-24T21:04:46.246018+0000","last_active":"2023-12-24T21:14:58.078520+0000","last_peered":"2023-12-24T21:14:58.078520+0000","last_clean":"2023-12-24T21:14:58.078520+0000","last_became_active":"2023-12-24T17:57:44.102598+0000","last_became_peered":"2023-12-24T17:57:44.102598+0000","last_unstale":"2023-12-24T21:14:58.078520+0000","last_undegraded":"2023-12-24T21:14:58.078520+0000","last_fullsized":"2023-12-24T21:14:58.078520+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 739s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035727099999999999,"stat_sum":{"num_bytes":265846967,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":817,"num_read_kb":7922,"num_write":2312,"num_write_kb":1104148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"159'5324","reported_seq":13014,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:18:38.408481+0000","last_change":"2023-12-24T21:18:38.408481+0000","last_active":"2023-12-24T21:18:38.408481+0000","last_peered":"2023-12-24T21:18:38.408481+0000","last_clean":"2023-12-24T21:18:38.408481+0000","last_became_active":"2023-12-24T17:57:51.125970+0000","last_became_peered":"2023-12-24T17:57:51.125970+0000","last_unstale":"2023-12-24T21:18:38.408481+0000","last_undegraded":"2023-12-24T21:18:38.408481+0000","last_fullsized":"2023-12-24T21:18:38.408481+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:50.110900+0000","last_clean_scrub_stamp":"2023-12-24T17:57:50.110900+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000355478,"stat_sum":{"num_bytes":68138159,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":35,"num_read":20172,"num_read_kb":105172,"num_write":4969,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1199,"num_flush":206,"num_flush_kb":26397,"num_evict":441,"num_evict_kb":22121,"num_promote":494,"num_flush_mode_high":0,"num_flush_mode_low":1,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"159'2044","reported_seq":3290,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:18:39.445513+0000","last_change":"2023-12-24T21:18:39.445513+0000","last_active":"2023-12-24T21:18:39.445513+0000","last_peered":"2023-12-24T21:18:39.445513+0000","last_clean":"2023-12-24T21:18:39.445513+0000","last_became_active":"2023-12-24T17:57:44.103977+0000","last_became_peered":"2023-12-24T17:57:44.103977+0000","last_unstale":"2023-12-24T21:18:39.445513+0000","last_undegraded":"2023-12-24T21:18:39.445513+0000","last_fullsized":"2023-12-24T21:18:39.445513+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2044,"log_dups_size":0,"ondisk_log_size":2044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040882899999999998,"stat_sum":{"num_bytes":242581608,"num_objects":71,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":695,"num_read_kb":14082,"num_write":1918,"num_write_kb":1087929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"159'2335","reported_seq":3319,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-12-24T21:14:59.111392+0000","last_change":"2023-12-24T18:02:38.125628+0000","last_active":"2023-12-24T21:14:59.111392+0000","last_peered":"2023-12-24T21:14:59.111392+0000","last_clean":"2023-12-24T21:14:59.111392+0000","last_became_active":"2023-12-24T17:57:44.102920+0000","last_became_peered":"2023-12-24T17:57:44.102920+0000","last_unstale":"2023-12-24T21:14:59.111392+0000","last_undegraded":"2023-12-24T21:14:59.111392+0000","last_fullsized":"2023-12-24T21:14:59.111392+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-24T17:57:43.091708+0000","last_clean_scrub_stamp":"2023-12-24T17:57:43.091708+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063372999999999999,"stat_sum":{"num_bytes":272937132,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":733,"num_read_kb":9769,"num_write":2187,"num_write_kb":1091173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":193262824,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":149,"num_read":62141,"num_read_kb":371959,"num_write":13291,"num_write_kb":10989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":18,"num_bytes_hit_set_archive":5156,"num_flush":801,"num_flush_kb":96114,"num_evict":1977,"num_evict_kb":185225,"num_promote":2178,"num_flush_mode_high":0,"num_flush_mode_low":2,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329547776,"data_stored":329394640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18539,"ondisk_log_size":18539,"up":8,"acting":8,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2039459609,"num_objects":565,"num_object_clones":1,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":1,"num_read":6075,"num_read_kb":81880,"num_write":16781,"num_write_kb":8938499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346669056,"data_stored":346439218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18753,"ondisk_log_size":18753,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359740787,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526468,"kb_used_data":220988,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93845372,"statfs":{"total":96636764160,"available":96097660928,"internally_reserved":0,"allocated":226291712,"data_stored":225519660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,4,0,4,6,12,6],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:18:27 2023","interfaces":[{"interface":"back","average":{"1min":0.23400000000000001,"5min":0.217,"15min":0.22},"min":{"1min":0.16,"5min":0.154,"15min":0.14399999999999999},"max":{"1min":0.40100000000000002,"5min":0.40500000000000003,"15min":1.198},"last":0.155},{"interface":"front","average":{"1min":0.25900000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.161},"max":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":1.1160000000000001},"last":0.22900000000000001}]},{"osd":1,"last update":"Sun Dec 24 21:18:27 2023","interfaces":[{"interface":"back","average":{"1min":0.27300000000000002,"5min":0.26200000000000001,"15min":0.255},"min":{"1min":0.19500000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.40400000000000003,"5min":1.1499999999999999,"15min":1.1639999999999999},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.28199999999999997,"5min":0.26900000000000002,"15min":0.26800000000000002},"min":{"1min":0.20300000000000001,"5min":0.186,"15min":0.16900000000000001},"max":{"1min":0.46100000000000002,"5min":0.46500000000000002,"15min":1.153},"last":0.26400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359740786,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526392,"kb_used_data":216816,"kb_used_omap":0,"kb_used_meta":309568,"kb_avail":93845448,"statfs":{"total":96636764160,"available":96097738752,"internally_reserved":0,"allocated":222019584,"data_stored":221260316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316997632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 24 21:18:27 2023","interfaces":[{"interface":"back","average":{"1min":0.19400000000000001,"5min":0.21099999999999999,"15min":0.20399999999999999},"min":{"1min":0.14399999999999999,"5min":0.14000000000000001,"15min":0.13900000000000001},"max":{"1min":0.39100000000000001,"5min":0.71599999999999997,"15min":1.1140000000000001},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.24099999999999999,"5min":0.23400000000000001,"15min":0.24199999999999999},"min":{"1min":0.17199999999999999,"5min":0.159,"15min":0.14799999999999999},"max":{"1min":0.437,"5min":0.63600000000000001,"15min":1.1120000000000001},"last":0.21199999999999999}]},{"osd":2,"last update":"Sun Dec 24 21:18:27 2023","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":0.26000000000000001,"15min":0.26700000000000002},"min":{"1min":0.20899999999999999,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":0.40999999999999998,"5min":0.66600000000000004,"15min":1.141},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.28899999999999998,"15min":0.29699999999999999},"min":{"1min":0.23000000000000001,"5min":0.19400000000000001,"15min":0.192},"max":{"1min":0.46400000000000002,"5min":0.68300000000000005,"15min":1.157},"last":0.26100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359740789,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533176,"kb_used_data":227696,"kb_used_omap":0,"kb_used_meta":305472,"kb_avail":93838664,"statfs":{"total":96636764160,"available":96090791936,"internally_reserved":0,"allocated":233160704,"data_stored":232337159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312803328},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,256],"upper_bound":33554432},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 24 21:18:29 2023","interfaces":[{"interface":"back","average":{"1min":0.20699999999999999,"5min":0.193,"15min":0.193},"min":{"1min":0.16200000000000001,"5min":0.14099999999999999,"15min":0.126},"max":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.437},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.23999999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"min":{"1min":0.19,"5min":0.17599999999999999,"15min":0.153},"max":{"1min":0.42399999999999999,"5min":0.46100000000000002,"15min":0.46100000000000002},"last":0.41699999999999998}]},{"osd":2,"last update":"Sun Dec 24 21:18:29 2023","interfaces":[{"interface":"back","average":{"1min":0.23699999999999999,"5min":0.22500000000000001,"15min":0.22900000000000001},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.157},"max":{"1min":0.47499999999999998,"5min":0.47599999999999998,"15min":0.63600000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.27900000000000003,"5min":0.26500000000000001,"15min":0.26700000000000002},"min":{"1min":0.23499999999999999,"5min":0.20599999999999999,"15min":0.192},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.626},"last":0.44}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113172480,"data_stored":113098116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173334528,"data_stored":173219609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":60162048,"data_stored":60121493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106897408,"data_stored":106870985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57876480,"data_stored":57826335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164773888,"data_stored":164697320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-24T21:18:48.238 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:48.238 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:48.238 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:48.238 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:48.238 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:48.239 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:48.239 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-24T17:57:37.923951+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=37, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:48.239 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:48.239 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:48.240 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:48.240 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-12-24T17:57:50.110900+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=50, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:48.240 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:48.240 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-24T17:57:43.091708+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=17, tm_min=57, tm_sec=43, tm_wday=6, tm_yday=358, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=24, tm_hour=21, tm_min=0, tm_sec=42, tm_wday=6, tm_yday=358, tm_isdst=0) 2023-12-24T21:18:48.241 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f/qa/tasks/ceph.py", line 1910, in task osd_scrub_pgs(ctx, config) File "/home/teuthworker/src/github.com_ceph_ceph-c_5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f/qa/tasks/ceph.py", line 1305, in osd_scrub_pgs raise RuntimeError('Exiting scrub checking -- not all pgs scrubbed.') RuntimeError: Exiting scrub checking -- not all pgs scrubbed. 2023-12-24T21:18:48.251 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-24T21:18:48.252 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-24T21:18:48.252 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-12-24T21:18:48.252 INFO:teuthology.orchestra.run:waiting for 300 2023-12-24T21:18:48.293 INFO:tasks.ceph.osd.0:Stopped 2023-12-24T21:18:48.294 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-12-24T21:18:48.294 INFO:teuthology.orchestra.run:waiting for 300 2023-12-24T21:18:48.342 INFO:tasks.ceph.osd.1:Stopped 2023-12-24T21:18:48.342 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-12-24T21:18:48.343 INFO:teuthology.orchestra.run:waiting for 300 2023-12-24T21:18:48.377 INFO:tasks.ceph.osd.2:Stopped 2023-12-24T21:18:48.377 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-24T21:18:48.378 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-12-24T21:18:48.378 INFO:teuthology.orchestra.run:waiting for 300 2023-12-24T21:18:48.441 INFO:tasks.ceph.mgr.x:Stopped 2023-12-24T21:18:48.441 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-24T21:18:48.442 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-12-24T21:18:48.442 INFO:teuthology.orchestra.run:waiting for 300 2023-12-24T21:18:48.498 INFO:tasks.ceph.mon.a:Stopped 2023-12-24T21:18:48.499 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-24T21:18:48.499 DEBUG:teuthology.orchestra.run.smithi087:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | head -n 1 2023-12-24T21:18:48.566 INFO:teuthology.orchestra.run.smithi087.stdout:1703440992.6423671 osd.0 (osd.0) 3 : cluster [WRN] 161 slow requests (by type [ 'reached pg' : 161 ] most affected pool [ 'cache' : 161 ]) 2023-12-24T21:18:48.568 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-12-24T21:18:48.569 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi087.front.sepia.ceph.com 2023-12-24T21:18:48.569 DEBUG:teuthology.orchestra.run.smithi087:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-12-24T21:18:48.988 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi087.front.sepia.ceph.com 2023-12-24T21:18:48.988 DEBUG:teuthology.orchestra.run.smithi087:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-12-24T21:18:49.093 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi087.front.sepia.ceph.com 2023-12-24T21:18:49.094 DEBUG:teuthology.orchestra.run.smithi087:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-12-24T21:18:49.176 INFO:tasks.ceph:Archiving mon data... 2023-12-24T21:18:49.178 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-12-24_15:40:05-rbd-wip-yuri4-testing-2023-12-21-0917-quincy-distro-default-smithi/7500245/data/mon.a.tgz 2023-12-24T21:18:49.178 DEBUG:teuthology.orchestra.run.smithi087:> mktemp 2023-12-24T21:18:49.197 INFO:teuthology.orchestra.run.smithi087.stdout:/tmp/tmp.ku9MsxOvYe 2023-12-24T21:18:49.197 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.ku9MsxOvYe 2023-12-24T21:18:49.357 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0666 /tmp/tmp.ku9MsxOvYe 2023-12-24T21:18:49.445 DEBUG:teuthology.orchestra.remote:smithi087:/tmp/tmp.ku9MsxOvYe is 416KB 2023-12-24T21:18:49.547 DEBUG:teuthology.orchestra.run.smithi087:> rm -fr /tmp/tmp.ku9MsxOvYe 2023-12-24T21:18:49.564 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-24T21:18:49.565 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-24T21:18:49.631 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-12-24T21:18:49.632 DEBUG:teuthology.orchestra.run.smithi087:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-12-24T21:18:49.699 INFO:teuthology.orchestra.run.smithi087.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-12-24T21:18:49.705 INFO:tasks.ceph:Archiving crash dumps... 2023-12-24T21:18:49.707 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-12-24_15:40:05-rbd-wip-yuri4-testing-2023-12-21-0917-quincy-distro-default-smithi/7500245/remote/smithi087/crash 2023-12-24T21:18:49.709 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-24T21:18:49.772 INFO:tasks.ceph:Compressing logs... 2023-12-24T21:18:49.772 DEBUG:teuthology.orchestra.run.smithi087:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-24T21:18:50.339 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156170.log 2023-12-24T21:18:50.339 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144843.log 2023-12-24T21:18:50.339 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135389.log 2023-12-24T21:18:50.339 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156170.log.gz 2023-12-24T21:18:50.340 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144843.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144843.log.gz 2023-12-24T21:18:50.340 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170631.log 2023-12-24T21:18:50.340 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137567.log 2023-12-24T21:18:50.340 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135389.log.gz 2023-12-24T21:18:50.340 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.170631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134545.log 2023-12-24T21:18:50.340 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170631.log.gz 2023-12-24T21:18:50.340 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154465.log 2023-12-24T21:18:50.340 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137567.log.gz 2023-12-24T21:18:50.341 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158472.log 2023-12-24T21:18:50.341 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134545.log.gz 2023-12-24T21:18:50.341 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128933.log 2023-12-24T21:18:50.341 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154465.log.gz 2023-12-24T21:18:50.341 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147300.log 2023-12-24T21:18:50.341 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158472.log.gz 2023-12-24T21:18:50.341 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128933.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.122292.log 2023-12-24T21:18:50.341 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.128933.log.gz 2023-12-24T21:18:50.341 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151661.log 2023-12-24T21:18:50.342 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147300.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147300.log.gz 2023-12-24T21:18:50.342 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140878.log 2023-12-24T21:18:50.342 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122292.log.gz 2023-12-24T21:18:50.342 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146285.log 2023-12-24T21:18:50.342 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151661.log.gz 2023-12-24T21:18:50.342 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161758.log 2023-12-24T21:18:50.342 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140878.log.gz 2023-12-24T21:18:50.342 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146285.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146285.log.gz 2023-12-24T21:18:50.342 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153476.log 2023-12-24T21:18:50.343 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131544.log 2023-12-24T21:18:50.343 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161758.log: 12.9% -- replaced with /var/log/ceph/ceph-client.admin.161758.log.gz 2023-12-24T21:18:50.343 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147776.log 2023-12-24T21:18:50.343 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153476.log.gz 2023-12-24T21:18:50.343 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145990.log 2023-12-24T21:18:50.343 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131544.log.gz 2023-12-24T21:18:50.343 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121888.log 2023-12-24T21:18:50.343 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147776.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147776.log.gz 2023-12-24T21:18:50.343 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139844.log 2023-12-24T21:18:50.344 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.145990.log.gz 2023-12-24T21:18:50.344 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132900.log 2023-12-24T21:18:50.344 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.121888.log.gz 2023-12-24T21:18:50.344 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129853.log 2023-12-24T21:18:50.344 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139844.log.gz 2023-12-24T21:18:50.344 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130439.log 2023-12-24T21:18:50.344 INFO:teuthology.orchestra.run.smithi087.stderr: 23.8% -- replaced with /var/log/ceph/ceph-client.admin.132900.log.gz 2023-12-24T21:18:50.344 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148963.log 2023-12-24T21:18:50.344 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129853.log.gz 2023-12-24T21:18:50.345 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157201.log 2023-12-24T21:18:50.345 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130439.log.gz 2023-12-24T21:18:50.345 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138938.log 2023-12-24T21:18:50.345 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148963.log.gz 2023-12-24T21:18:50.345 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122041.log 2023-12-24T21:18:50.345 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157201.log.gz 2023-12-24T21:18:50.345 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163869.log 2023-12-24T21:18:50.345 INFO:teuthology.orchestra.run.smithi087.stderr: 23.5% -- replaced with /var/log/ceph/ceph-client.admin.138938.log.gz 2023-12-24T21:18:50.345 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161507.log 2023-12-24T21:18:50.346 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122041.log.gz 2023-12-24T21:18:50.346 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142267.log 2023-12-24T21:18:50.346 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163869.log.gz 2023-12-24T21:18:50.346 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142591.log 2023-12-24T21:18:50.346 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161507.log.gz 2023-12-24T21:18:50.346 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128633.log 2023-12-24T21:18:50.346 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142267.log.gz 2023-12-24T21:18:50.346 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144370.log 2023-12-24T21:18:50.346 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142591.log.gz 2023-12-24T21:18:50.347 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161721.log 2023-12-24T21:18:50.347 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128633.log.gz 2023-12-24T21:18:50.347 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142487.log 2023-12-24T21:18:50.347 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144370.log.gz 2023-12-24T21:18:50.347 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124133.log 2023-12-24T21:18:50.347 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161721.log.gz 2023-12-24T21:18:50.347 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158038.log 2023-12-24T21:18:50.347 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142487.log.gz 2023-12-24T21:18:50.347 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142447.log 2023-12-24T21:18:50.348 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124133.log.gz 2023-12-24T21:18:50.348 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157038.log 2023-12-24T21:18:50.348 INFO:teuthology.orchestra.run.smithi087.stderr: 51.5% -- replaced with /var/log/ceph/ceph-client.admin.158038.log.gz 2023-12-24T21:18:50.348 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133719.log 2023-12-24T21:18:50.348 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142447.log.gz 2023-12-24T21:18:50.348 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134954.log 2023-12-24T21:18:50.348 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157038.log.gz 2023-12-24T21:18:50.348 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126840.log 2023-12-24T21:18:50.348 INFO:teuthology.orchestra.run.smithi087.stderr: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.133719.log.gz 2023-12-24T21:18:50.349 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156779.log 2023-12-24T21:18:50.349 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134954.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134954.log.gz 2023-12-24T21:18:50.349 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129273.log 2023-12-24T21:18:50.349 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.126840.log.gz 2023-12-24T21:18:50.349 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166985.log 2023-12-24T21:18:50.349 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156779.log.gz 2023-12-24T21:18:50.349 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145840.log 2023-12-24T21:18:50.349 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129273.log.gz 2023-12-24T21:18:50.349 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.166985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136896.log 2023-12-24T21:18:50.350 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166985.log.gz 2023-12-24T21:18:50.350 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134097.log 2023-12-24T21:18:50.350 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145840.log.gz 2023-12-24T21:18:50.350 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139264.log 2023-12-24T21:18:50.350 INFO:teuthology.orchestra.run.smithi087.stderr: 56.6% -- replaced with /var/log/ceph/ceph-client.admin.136896.log.gz 2023-12-24T21:18:50.350 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164041.log 2023-12-24T21:18:50.350 INFO:teuthology.orchestra.run.smithi087.stderr: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.134097.log.gz 2023-12-24T21:18:50.350 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144134.log 2023-12-24T21:18:50.350 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139264.log.gz 2023-12-24T21:18:50.351 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.164041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154358.log 2023-12-24T21:18:50.351 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164041.log.gz 2023-12-24T21:18:50.351 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144134.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144134.log.gz 2023-12-24T21:18:50.351 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131102.log 2023-12-24T21:18:50.351 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158628.log 2023-12-24T21:18:50.351 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154358.log.gz 2023-12-24T21:18:50.351 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135773.log 2023-12-24T21:18:50.351 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131102.log.gz 2023-12-24T21:18:50.351 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153304.log 2023-12-24T21:18:50.352 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158628.log.gz 2023-12-24T21:18:50.352 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126627.log 2023-12-24T21:18:50.352 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135773.log.gz 2023-12-24T21:18:50.352 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161945.log 2023-12-24T21:18:50.352 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153304.log.gz 2023-12-24T21:18:50.352 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154444.log 2023-12-24T21:18:50.352 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126627.log.gz 2023-12-24T21:18:50.352 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122426.log 2023-12-24T21:18:50.352 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161945.log.gz 2023-12-24T21:18:50.353 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147362.log 2023-12-24T21:18:50.353 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154444.log.gz 2023-12-24T21:18:50.353 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130881.log 2023-12-24T21:18:50.353 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122426.log.gz 2023-12-24T21:18:50.353 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157139.log 2023-12-24T21:18:50.353 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147362.log.gz 2023-12-24T21:18:50.353 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148468.log 2023-12-24T21:18:50.353 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130881.log.gz 2023-12-24T21:18:50.353 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126976.log 2023-12-24T21:18:50.354 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157139.log.gz 2023-12-24T21:18:50.354 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151876.log 2023-12-24T21:18:50.354 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148468.log.gz 2023-12-24T21:18:50.354 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141355.log 2023-12-24T21:18:50.354 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.126976.log.gz 2023-12-24T21:18:50.354 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154883.log 2023-12-24T21:18:50.354 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151876.log.gz 2023-12-24T21:18:50.354 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141355.log: 0.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.141355.log.gz--verbose 2023-12-24T21:18:50.354 INFO:teuthology.orchestra.run.smithi087.stderr:-- /var/log/ceph/ceph-client.admin.160367.log 2023-12-24T21:18:50.355 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127520.log 2023-12-24T21:18:50.355 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154883.log.gz 2023-12-24T21:18:50.355 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172476.log 2023-12-24T21:18:50.355 INFO:teuthology.orchestra.run.smithi087.stderr: 28.1% -- replaced with /var/log/ceph/ceph-client.admin.160367.log.gz 2023-12-24T21:18:50.355 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123811.log 2023-12-24T21:18:50.355 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127520.log.gz 2023-12-24T21:18:50.355 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.172476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162110.log 2023-12-24T21:18:50.355 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.172476.log.gz 2023-12-24T21:18:50.355 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121133.log 2023-12-24T21:18:50.356 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123811.log.gz 2023-12-24T21:18:50.356 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161206.log 2023-12-24T21:18:50.356 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162110.log.gz 2023-12-24T21:18:50.356 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134244.log 2023-12-24T21:18:50.356 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121133.log.gz 2023-12-24T21:18:50.356 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155213.log 2023-12-24T21:18:50.356 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161206.log.gz 2023-12-24T21:18:50.356 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134244.log: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.134244.log.gz 2023-12-24T21:18:50.357 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137692.log 2023-12-24T21:18:50.357 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129373.log 2023-12-24T21:18:50.357 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155213.log.gz 2023-12-24T21:18:50.357 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136974.log 2023-12-24T21:18:50.357 INFO:teuthology.orchestra.run.smithi087.stderr: 57.2% -- replaced with /var/log/ceph/ceph-client.admin.137692.log.gz 2023-12-24T21:18:50.357 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161528.log 2023-12-24T21:18:50.357 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129373.log.gz 2023-12-24T21:18:50.357 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123849.log 2023-12-24T21:18:50.357 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136974.log.gz 2023-12-24T21:18:50.358 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154573.log 2023-12-24T21:18:50.358 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161528.log.gz 2023-12-24T21:18:50.358 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130053.log 2023-12-24T21:18:50.358 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123849.log.gz 2023-12-24T21:18:50.358 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130093.log 2023-12-24T21:18:50.358 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154573.log.gz 2023-12-24T21:18:50.358 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133446.log 2023-12-24T21:18:50.358 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130053.log.gz 2023-12-24T21:18:50.358 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159946.log 2023-12-24T21:18:50.359 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130093.log.gz 2023-12-24T21:18:50.359 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140299.log 2023-12-24T21:18:50.359 INFO:teuthology.orchestra.run.smithi087.stderr: 23.1% -- replaced with /var/log/ceph/ceph-client.admin.133446.log.gz 2023-12-24T21:18:50.359 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156662.log 2023-12-24T21:18:50.359 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159946.log.gz 2023-12-24T21:18:50.359 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162410.log 2023-12-24T21:18:50.359 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140299.log.gz 2023-12-24T21:18:50.359 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136053.log 2023-12-24T21:18:50.359 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156662.log.gz 2023-12-24T21:18:50.360 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154186.log 2023-12-24T21:18:50.360 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162410.log.gz 2023-12-24T21:18:50.360 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124562.log 2023-12-24T21:18:50.360 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136053.log.gz 2023-12-24T21:18:50.360 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126648.log 2023-12-24T21:18:50.360 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154186.log.gz 2023-12-24T21:18:50.360 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125379.log 2023-12-24T21:18:50.360 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124562.log.gz 2023-12-24T21:18:50.360 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162469.log 2023-12-24T21:18:50.361 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126648.log.gz 2023-12-24T21:18:50.361 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132606.log 2023-12-24T21:18:50.361 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125379.log.gz 2023-12-24T21:18:50.361 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149710.log 2023-12-24T21:18:50.361 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162469.log.gz 2023-12-24T21:18:50.361 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160625.log 2023-12-24T21:18:50.361 INFO:teuthology.orchestra.run.smithi087.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.132606.log.gz 2023-12-24T21:18:50.361 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148683.log 2023-12-24T21:18:50.362 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149710.log.gz 2023-12-24T21:18:50.362 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156410.log 2023-12-24T21:18:50.362 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160625.log.gz 2023-12-24T21:18:50.362 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125875.log 2023-12-24T21:18:50.362 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148683.log.gz 2023-12-24T21:18:50.362 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128433.log 2023-12-24T21:18:50.362 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156410.log.gz 2023-12-24T21:18:50.362 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153627.log 2023-12-24T21:18:50.362 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125875.log.gz 2023-12-24T21:18:50.363 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153734.log 2023-12-24T21:18:50.363 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128433.log.gz 2023-12-24T21:18:50.363 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162727.log 2023-12-24T21:18:50.363 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153627.log.gz 2023-12-24T21:18:50.363 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125552.log 2023-12-24T21:18:50.363 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153734.log.gz 2023-12-24T21:18:50.363 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171921.log 2023-12-24T21:18:50.363 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162727.log.gz 2023-12-24T21:18:50.363 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126605.log 2023-12-24T21:18:50.364 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125552.log.gz 2023-12-24T21:18:50.364 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.171921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141176.log 2023-12-24T21:18:50.364 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171921.log.gz 2023-12-24T21:18:50.364 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126605.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126605.log.gz 2023-12-24T21:18:50.364 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131136.log 2023-12-24T21:18:50.364 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141176.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141176.log.gz 2023-12-24T21:18:50.364 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127214.log 2023-12-24T21:18:50.364 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155873.log 2023-12-24T21:18:50.365 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131136.log.gz 2023-12-24T21:18:50.365 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145009.log 2023-12-24T21:18:50.365 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127214.log.gz 2023-12-24T21:18:50.365 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146306.log 2023-12-24T21:18:50.365 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155873.log.gz 2023-12-24T21:18:50.365 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142961.log 2023-12-24T21:18:50.365 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145009.log.gz 2023-12-24T21:18:50.365 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121851.log 2023-12-24T21:18:50.365 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146306.log.gz 2023-12-24T21:18:50.366 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159966.log 2023-12-24T21:18:50.366 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142961.log.gz 2023-12-24T21:18:50.366 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121961.log 2023-12-24T21:18:50.366 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121851.log.gz 2023-12-24T21:18:50.366 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123714.log 2023-12-24T21:18:50.366 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159966.log.gz 2023-12-24T21:18:50.366 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146178.log 2023-12-24T21:18:50.366 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.121961.log.gz 2023-12-24T21:18:50.366 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139715.log 2023-12-24T21:18:50.367 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.123714.log.gz 2023-12-24T21:18:50.367 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136834.log 2023-12-24T21:18:50.367 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146178.log.gz 2023-12-24T21:18:50.367 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142550.log 2023-12-24T21:18:50.367 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139715.log.gz 2023-12-24T21:18:50.367 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168324.log 2023-12-24T21:18:50.367 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136834.log.gz 2023-12-24T21:18:50.367 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140235.log 2023-12-24T21:18:50.367 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142550.log.gz 2023-12-24T21:18:50.368 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.168324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134696.log 2023-12-24T21:18:50.368 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168324.log.gz 2023-12-24T21:18:50.368 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128030.log 2023-12-24T21:18:50.368 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140235.log.gz 2023-12-24T21:18:50.368 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150904.log 2023-12-24T21:18:50.368 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134696.log.gz 2023-12-24T21:18:50.368 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148597.log 2023-12-24T21:18:50.368 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.128030.log.gz 2023-12-24T21:18:50.369 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127605.log 2023-12-24T21:18:50.369 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150904.log.gz 2023-12-24T21:18:50.369 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149221.log 2023-12-24T21:18:50.369 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148597.log.gz 2023-12-24T21:18:50.369 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143025.log 2023-12-24T21:18:50.369 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127605.log.gz 2023-12-24T21:18:50.369 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141772.log 2023-12-24T21:18:50.369 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149221.log.gz 2023-12-24T21:18:50.369 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137077.log 2023-12-24T21:18:50.370 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143025.log.gz 2023-12-24T21:18:50.370 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142227.log 2023-12-24T21:18:50.370 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141772.log.gz 2023-12-24T21:18:50.370 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129293.log 2023-12-24T21:18:50.370 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137077.log.gz 2023-12-24T21:18:50.370 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147342.log 2023-12-24T21:18:50.370 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142227.log.gz 2023-12-24T21:18:50.370 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146371.log 2023-12-24T21:18:50.371 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129293.log.gz 2023-12-24T21:18:50.371 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114168.log 2023-12-24T21:18:50.371 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147342.log.gz 2023-12-24T21:18:50.371 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137899.log 2023-12-24T21:18:50.371 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146371.log.gz 2023-12-24T21:18:50.371 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.114168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169079.log 2023-12-24T21:18:50.371 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114168.log.gz 2023-12-24T21:18:50.371 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160047.log 2023-12-24T21:18:50.371 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137899.log.gz 2023-12-24T21:18:50.372 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.169079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151065.log 2023-12-24T21:18:50.372 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169079.log.gz 2023-12-24T21:18:50.372 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128573.log 2023-12-24T21:18:50.372 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160047.log.gz 2023-12-24T21:18:50.372 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151065.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151065.log.gz 2023-12-24T21:18:50.372 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119770.log 2023-12-24T21:18:50.372 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142008.log 2023-12-24T21:18:50.372 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128573.log.gz 2023-12-24T21:18:50.372 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.119770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155277.log 2023-12-24T21:18:50.373 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119770.log.gz 2023-12-24T21:18:50.373 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152193.log 2023-12-24T21:18:50.373 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142008.log.gz 2023-12-24T21:18:50.373 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146094.log 2023-12-24T21:18:50.373 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155277.log.gz 2023-12-24T21:18:50.373 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168226.log 2023-12-24T21:18:50.373 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152193.log.gz 2023-12-24T21:18:50.373 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163826.log 2023-12-24T21:18:50.373 INFO:teuthology.orchestra.run.smithi087.stderr: 55.4% -- replaced with /var/log/ceph/ceph-client.admin.146094.log.gz 2023-12-24T21:18:50.374 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.168226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153079.log 2023-12-24T21:18:50.374 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168226.log.gz 2023-12-24T21:18:50.374 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145175.log 2023-12-24T21:18:50.374 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163826.log.gz 2023-12-24T21:18:50.374 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133026.log 2023-12-24T21:18:50.374 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153079.log.gz 2023-12-24T21:18:50.374 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126541.log 2023-12-24T21:18:50.374 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145175.log.gz 2023-12-24T21:18:50.374 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124068.log 2023-12-24T21:18:50.375 INFO:teuthology.orchestra.run.smithi087.stderr: 23.1% -- replaced with /var/log/ceph/ceph-client.admin.133026.log.gz 2023-12-24T21:18:50.375 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156267.log 2023-12-24T21:18:50.375 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126541.log.gz 2023-12-24T21:18:50.375 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158172.log 2023-12-24T21:18:50.375 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124068.log.gz 2023-12-24T21:18:50.375 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137855.log 2023-12-24T21:18:50.375 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156267.log.gz 2023-12-24T21:18:50.375 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128733.log 2023-12-24T21:18:50.375 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158172.log.gz 2023-12-24T21:18:50.376 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160112.log 2023-12-24T21:18:50.376 INFO:teuthology.orchestra.run.smithi087.stderr: 57.0% -- replaced with /var/log/ceph/ceph-client.admin.137855.log.gz 2023-12-24T21:18:50.376 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140912.log 2023-12-24T21:18:50.376 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128733.log.gz 2023-12-24T21:18:50.376 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149629.log 2023-12-24T21:18:50.376 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160112.log.gz 2023-12-24T21:18:50.376 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143248.log 2023-12-24T21:18:50.376 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140912.log.gz 2023-12-24T21:18:50.376 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151575.log 2023-12-24T21:18:50.377 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149629.log.gz 2023-12-24T21:18:50.377 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154830.log 2023-12-24T21:18:50.377 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143248.log.gz 2023-12-24T21:18:50.377 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141944.log 2023-12-24T21:18:50.377 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151575.log.gz 2023-12-24T21:18:50.377 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125013.log 2023-12-24T21:18:50.377 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154830.log.gz 2023-12-24T21:18:50.377 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156368.log 2023-12-24T21:18:50.378 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141944.log.gz 2023-12-24T21:18:50.378 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170729.log 2023-12-24T21:18:50.378 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125013.log.gz 2023-12-24T21:18:50.378 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164304.log 2023-12-24T21:18:50.378 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156368.log.gz 2023-12-24T21:18:50.378 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.170729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128081.log 2023-12-24T21:18:50.378 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170729.log.gz 2023-12-24T21:18:50.378 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.164304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159741.log 2023-12-24T21:18:50.378 INFO:teuthology.orchestra.run.smithi087.stderr: 10.9% -- replaced with /var/log/ceph/ceph-client.admin.164304.log.gz 2023-12-24T21:18:50.379 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130033.log 2023-12-24T21:18:50.379 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.128081.log.gz 2023-12-24T21:18:50.379 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126198.log 2023-12-24T21:18:50.379 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159741.log.gz 2023-12-24T21:18:50.379 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130422.log 2023-12-24T21:18:50.379 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130033.log.gz 2023-12-24T21:18:50.379 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136117.log 2023-12-24T21:18:50.379 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126198.log.gz 2023-12-24T21:18:50.379 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144779.log 2023-12-24T21:18:50.380 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130422.log.gz 2023-12-24T21:18:50.380 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137015.log 2023-12-24T21:18:50.380 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136117.log.gz 2023-12-24T21:18:50.380 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148833.log 2023-12-24T21:18:50.380 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144779.log.gz 2023-12-24T21:18:50.380 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122236.log 2023-12-24T21:18:50.380 INFO:teuthology.orchestra.run.smithi087.stderr: 53.1% -- replaced with /var/log/ceph/ceph-client.admin.137015.log.gz 2023-12-24T21:18:50.380 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130558.log 2023-12-24T21:18:50.381 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148833.log.gz 2023-12-24T21:18:50.381 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126993.log 2023-12-24T21:18:50.381 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122236.log.gz 2023-12-24T21:18:50.381 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125357.log 2023-12-24T21:18:50.381 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130558.log.gz 2023-12-24T21:18:50.381 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145905.log 2023-12-24T21:18:50.381 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.126993.log.gz 2023-12-24T21:18:50.381 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131748.log 2023-12-24T21:18:50.382 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125357.log.gz 2023-12-24T21:18:50.382 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163934.log 2023-12-24T21:18:50.382 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145905.log.gz 2023-12-24T21:18:50.382 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151811.log 2023-12-24T21:18:50.382 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131748.log.gz 2023-12-24T21:18:50.382 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148704.log 2023-12-24T21:18:50.382 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163934.log.gz 2023-12-24T21:18:50.382 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156472.log 2023-12-24T21:18:50.382 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151811.log.gz 2023-12-24T21:18:50.383 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155914.log 2023-12-24T21:18:50.383 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148704.log.gz 2023-12-24T21:18:50.383 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160324.log 2023-12-24T21:18:50.383 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156472.log.gz 2023-12-24T21:18:50.383 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132837.log 2023-12-24T21:18:50.383 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155914.log.gz 2023-12-24T21:18:50.383 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127197.log 2023-12-24T21:18:50.383 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160324.log.gz 2023-12-24T21:18:50.383 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152917.log 2023-12-24T21:18:50.384 INFO:teuthology.orchestra.run.smithi087.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.132837.log.gz 2023-12-24T21:18:50.384 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155822.log 2023-12-24T21:18:50.384 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127197.log.gz 2023-12-24T21:18:50.384 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137153.log 2023-12-24T21:18:50.384 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152917.log.gz 2023-12-24T21:18:50.384 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121649.log 2023-12-24T21:18:50.384 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155822.log.gz 2023-12-24T21:18:50.384 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125940.log 0.0% 2023-12-24T21:18:50.384 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.137153.log.gz 2023-12-24T21:18:50.385 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153059.log 2023-12-24T21:18:50.385 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121649.log.gz 2023-12-24T21:18:50.385 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155744.log 2023-12-24T21:18:50.385 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125940.log.gz 2023-12-24T21:18:50.385 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153059.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.153059.log.gz-5 2023-12-24T21:18:50.385 INFO:teuthology.orchestra.run.smithi087.stderr:--verbose -- /var/log/ceph/ceph-client.admin.139100.log 2023-12-24T21:18:50.385 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155744.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155744.log.gz 2023-12-24T21:18:50.385 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123918.log 2023-12-24T21:18:50.385 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121348.log 2023-12-24T21:18:50.386 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139100.log.gz 2023-12-24T21:18:50.386 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144650.log 2023-12-24T21:18:50.386 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123918.log.gz 2023-12-24T21:18:50.386 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150776.log 2023-12-24T21:18:50.386 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121348.log.gz 2023-12-24T21:18:50.386 INFO:teuthology.orchestra.run.smithi087.stderr:gzip /var/log/ceph/ceph-client.admin.144650.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.138738.log 2023-12-24T21:18:50.386 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144650.log.gz 2023-12-24T21:18:50.386 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157921.log 2023-12-24T21:18:50.387 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150776.log.gz 2023-12-24T21:18:50.387 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154422.log 2023-12-24T21:18:50.387 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138738.log.gz 2023-12-24T21:18:50.387 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157238.log 2023-12-24T21:18:50.387 INFO:teuthology.orchestra.run.smithi087.stderr: 53.5% -- replaced with /var/log/ceph/ceph-client.admin.157921.log.gz 2023-12-24T21:18:50.387 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143510.log 2023-12-24T21:18:50.387 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154422.log.gz 2023-12-24T21:18:50.387 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154207.log 2023-12-24T21:18:50.387 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157238.log.gz 2023-12-24T21:18:50.388 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136771.log 2023-12-24T21:18:50.388 INFO:teuthology.orchestra.run.smithi087.stderr: 57.2% -- replaced with /var/log/ceph/ceph-client.admin.143510.log.gz 2023-12-24T21:18:50.388 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154745.log 2023-12-24T21:18:50.388 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154207.log.gz 2023-12-24T21:18:50.388 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139479.log 2023-12-24T21:18:50.388 INFO:teuthology.orchestra.run.smithi087.stderr: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.136771.log.gz 2023-12-24T21:18:50.388 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123211.log 2023-12-24T21:18:50.388 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154745.log.gz 2023-12-24T21:18:50.388 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135287.log 2023-12-24T21:18:50.389 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139479.log.gz 2023-12-24T21:18:50.389 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141158.log 2023-12-24T21:18:50.389 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123211.log.gz 2023-12-24T21:18:50.389 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119427.log 2023-12-24T21:18:50.389 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135287.log.gz 2023-12-24T21:18:50.389 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141158.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141158.log.gz 2023-12-24T21:18:50.389 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131425.log 2023-12-24T21:18:50.389 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.119427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125035.log 2023-12-24T21:18:50.389 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119427.log.gz 2023-12-24T21:18:50.390 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149178.log 2023-12-24T21:18:50.390 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131425.log.gz 2023-12-24T21:18:50.390 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156723.log 2023-12-24T21:18:50.390 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125035.log.gz 2023-12-24T21:18:50.390 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136855.log 2023-12-24T21:18:50.390 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149178.log.gz 2023-12-24T21:18:50.390 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161965.log 2023-12-24T21:18:50.390 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156723.log.gz 2023-12-24T21:18:50.390 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168373.log 2023-12-24T21:18:50.391 INFO:teuthology.orchestra.run.smithi087.stderr: 58.8% -- replaced with /var/log/ceph/ceph-client.admin.136855.log.gz 2023-12-24T21:18:50.391 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143875.log 2023-12-24T21:18:50.391 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161965.log.gz 2023-12-24T21:18:50.391 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.168373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161700.log 2023-12-24T21:18:50.391 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168373.log.gz 2023-12-24T21:18:50.391 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137115.log 2023-12-24T21:18:50.391 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143875.log.gz 2023-12-24T21:18:50.391 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163165.log 2023-12-24T21:18:50.391 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161700.log.gz 2023-12-24T21:18:50.392 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126327.log 2023-12-24T21:18:50.392 INFO:teuthology.orchestra.run.smithi087.stderr: 15.7% -- replaced with /var/log/ceph/ceph-client.admin.137115.log.gz 2023-12-24T21:18:50.392 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130677.log 2023-12-24T21:18:50.392 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163165.log.gz 2023-12-24T21:18:50.392 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160582.log 2023-12-24T21:18:50.392 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126327.log.gz 2023-12-24T21:18:50.392 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149794.log 2023-12-24T21:18:50.392 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130677.log.gz 2023-12-24T21:18:50.392 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139586.log 2023-12-24T21:18:50.393 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160582.log.gz 2023-12-24T21:18:50.393 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149794.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149794.log.gz 2023-12-24T21:18:50.393 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144987.log 2023-12-24T21:18:50.393 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134181.log 2023-12-24T21:18:50.393 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139586.log.gz 2023-12-24T21:18:50.393 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153670.log 2023-12-24T21:18:50.393 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144987.log.gz 2023-12-24T21:18:50.393 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155001.log 2023-12-24T21:18:50.393 INFO:teuthology.orchestra.run.smithi087.stderr: 23.4% -- replaced with /var/log/ceph/ceph-client.admin.134181.log.gz 2023-12-24T21:18:50.394 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147181.log 2023-12-24T21:18:50.394 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153670.log.gz 2023-12-24T21:18:50.394 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140844.log 2023-12-24T21:18:50.394 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155001.log.gz 2023-12-24T21:18:50.394 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127316.log 2023-12-24T21:18:50.394 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147181.log.gz 2023-12-24T21:18:50.394 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140844.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140844.log.gz 2023-12-24T21:18:50.394 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130253.log 2023-12-24T21:18:50.395 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169929.log 2023-12-24T21:18:50.395 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127316.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127316.log.gz 2023-12-24T21:18:50.395 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134330.log 2023-12-24T21:18:50.395 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130253.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130253.log.gz 2023-12-24T21:18:50.395 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.169929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147078.log 2023-12-24T21:18:50.395 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169929.log.gz 2023-12-24T21:18:50.395 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138879.log 2023-12-24T21:18:50.395 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134330.log.gz 2023-12-24T21:18:50.395 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152545.log 2023-12-24T21:18:50.396 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147078.log.gz 2023-12-24T21:18:50.396 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142755.log 2023-12-24T21:18:50.396 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138879.log.gz 2023-12-24T21:18:50.396 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123697.log 2023-12-24T21:18:50.396 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152545.log.gz 2023-12-24T21:18:50.396 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124691.log 2023-12-24T21:18:50.396 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142755.log.gz 2023-12-24T21:18:50.396 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152365.log 2023-12-24T21:18:50.396 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123697.log.gz 2023-12-24T21:18:50.397 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133278.log 2023-12-24T21:18:50.397 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124691.log.gz 2023-12-24T21:18:50.397 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141965.log 2023-12-24T21:18:50.397 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152365.log.gz 2023-12-24T21:18:50.397 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121284.log 2023-12-24T21:18:50.397 INFO:teuthology.orchestra.run.smithi087.stderr: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.133278.log.gz 2023-12-24T21:18:50.397 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163294.log 2023-12-24T21:18:50.397 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141965.log.gz 2023-12-24T21:18:50.397 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148103.log 2023-12-24T21:18:50.398 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121284.log.gz 2023-12-24T21:18:50.398 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149856.log 2023-12-24T21:18:50.398 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163294.log.gz 2023-12-24T21:18:50.398 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145733.log 2023-12-24T21:18:50.398 INFO:teuthology.orchestra.run.smithi087.stderr: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.148103.log.gz 2023-12-24T21:18:50.398 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149856.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149856.log.gz 2023-12-24T21:18:50.398 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127826.log 2023-12-24T21:18:50.398 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155154.log 2023-12-24T21:18:50.398 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145733.log.gz 2023-12-24T21:18:50.399 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168079.log 2023-12-24T21:18:50.399 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127826.log.gz 2023-12-24T21:18:50.399 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160604.log 2023-12-24T21:18:50.399 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155154.log.gz 2023-12-24T21:18:50.399 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.168079.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168079.log.gz 2023-12-24T21:18:50.399 INFO:teuthology.orchestra.run.smithi087.stderr:--verbose -- /var/log/ceph/ceph-client.admin.143982.log 2023-12-24T21:18:50.399 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123961.log 2023-12-24T21:18:50.399 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160604.log.gz 2023-12-24T21:18:50.399 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157786.log 2023-12-24T21:18:50.400 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143982.log.gz 2023-12-24T21:18:50.400 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138440.log 2023-12-24T21:18:50.400 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123961.log.gz 2023-12-24T21:18:50.400 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156585.log 2023-12-24T21:18:50.400 INFO:teuthology.orchestra.run.smithi087.stderr: 56.4% -- replaced with /var/log/ceph/ceph-client.admin.157786.log.gz 2023-12-24T21:18:50.400 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166829.log 2023-12-24T21:18:50.400 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138440.log.gz 2023-12-24T21:18:50.400 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151403.log 2023-12-24T21:18:50.400 INFO:teuthology.orchestra.run.smithi087.stderr: 53.3% -- replaced with /var/log/ceph/ceph-client.admin.156585.log.gz 2023-12-24T21:18:50.401 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.166829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150279.log 2023-12-24T21:18:50.401 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166829.log.gz 2023-12-24T21:18:50.401 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153498.log 2023-12-24T21:18:50.401 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151403.log.gz 2023-12-24T21:18:50.401 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157706.log 2023-12-24T21:18:50.401 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150279.log.gz 2023-12-24T21:18:50.401 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130113.log 2023-12-24T21:18:50.401 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153498.log.gz 2023-12-24T21:18:50.402 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160303.log 2023-12-24T21:18:50.402 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157706.log.gz 2023-12-24T21:18:50.402 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147901.log 2023-12-24T21:18:50.402 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130113.log.gz 2023-12-24T21:18:50.402 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148769.log 2023-12-24T21:18:50.402 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160303.log.gz 2023-12-24T21:18:50.402 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142368.log 2023-12-24T21:18:50.402 INFO:teuthology.orchestra.run.smithi087.stderr: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.147901.log.gz 2023-12-24T21:18:50.402 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168422.log 2023-12-24T21:18:50.403 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148769.log.gz 2023-12-24T21:18:50.403 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142368.log: gzip -5 --verbose 0.0%-- /var/log/ceph/ceph-client.admin.126891.log -- replaced with /var/log/ceph/ceph-client.admin.142368.log.gz 2023-12-24T21:18:50.403 INFO:teuthology.orchestra.run.smithi087.stderr: 2023-12-24T21:18:50.403 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.168422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160195.log 2023-12-24T21:18:50.403 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168422.log.gz 2023-12-24T21:18:50.403 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122350.log 2023-12-24T21:18:50.403 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.126891.log.gz 2023-12-24T21:18:50.403 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128166.log 2023-12-24T21:18:50.403 INFO:teuthology.orchestra.run.smithi087.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.160195.log.gz 2023-12-24T21:18:50.404 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143703.log 2023-12-24T21:18:50.404 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122350.log.gz 2023-12-24T21:18:50.404 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141514.log 2023-12-24T21:18:50.404 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.128166.log.gz 2023-12-24T21:18:50.404 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143703.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.143703.log.gz-5 2023-12-24T21:18:50.404 INFO:teuthology.orchestra.run.smithi087.stderr:--verbose -- /var/log/ceph/ceph-client.admin.126391.log 2023-12-24T21:18:50.404 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142094.log 2023-12-24T21:18:50.404 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141514.log.gz 2023-12-24T21:18:50.405 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145776.log 2023-12-24T21:18:50.405 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126391.log.gz 2023-12-24T21:18:50.405 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142094.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142094.log.gz 2023-12-24T21:18:50.405 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159151.log 2023-12-24T21:18:50.405 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121999.log 2023-12-24T21:18:50.405 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145776.log.gz 2023-12-24T21:18:50.405 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161442.log 2023-12-24T21:18:50.405 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159151.log.gz 2023-12-24T21:18:50.405 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127112.log 2023-12-24T21:18:50.406 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.121999.log.gz 2023-12-24T21:18:50.406 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140256.log 2023-12-24T21:18:50.406 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161442.log.gz 2023-12-24T21:18:50.406 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160668.log 2023-12-24T21:18:50.406 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127112.log.gz 2023-12-24T21:18:50.406 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142116.log 2023-12-24T21:18:50.406 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140256.log.gz 2023-12-24T21:18:50.406 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128453.log 2023-12-24T21:18:50.406 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160668.log.gz 2023-12-24T21:18:50.407 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124004.log 2023-12-24T21:18:50.407 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142116.log.gz 2023-12-24T21:18:50.407 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131595.log 2023-12-24T21:18:50.407 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128453.log.gz 2023-12-24T21:18:50.407 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124734.log 2023-12-24T21:18:50.407 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124004.log.gz 2023-12-24T21:18:50.407 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134139.log 2023-12-24T21:18:50.407 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131595.log.gz 2023-12-24T21:18:50.407 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153185.log 2023-12-24T21:18:50.408 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124734.log.gz 2023-12-24T21:18:50.408 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122953.log 2023-12-24T21:18:50.408 INFO:teuthology.orchestra.run.smithi087.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.134139.log.gz 2023-12-24T21:18:50.408 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154164.log 2023-12-24T21:18:50.408 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153185.log.gz 2023-12-24T21:18:50.408 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143575.log 2023-12-24T21:18:50.408 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122953.log.gz 2023-12-24T21:18:50.408 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148339.log 2023-12-24T21:18:50.408 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154164.log.gz 2023-12-24T21:18:50.409 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160153.log 2023-12-24T21:18:50.409 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143575.log.gz 2023-12-24T21:18:50.409 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148339.log: /var/log/ceph/ceph-client.admin.136696.log 2023-12-24T21:18:50.409 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148339.log.gz 2023-12-24T21:18:50.409 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131646.log 2023-12-24T21:18:50.409 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160153.log.gz 2023-12-24T21:18:50.409 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147139.log 2023-12-24T21:18:50.409 INFO:teuthology.orchestra.run.smithi087.stderr: 18.5% -- replaced with /var/log/ceph/ceph-client.admin.136696.log.gz 2023-12-24T21:18:50.410 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135924.log 2023-12-24T21:18:50.410 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131646.log.gz 2023-12-24T21:18:50.410 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172419.log 2023-12-24T21:18:50.410 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147139.log.gz 2023-12-24T21:18:50.410 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135924.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.122996.log 2023-12-24T21:18:50.410 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.135924.log.gz 2023-12-24T21:18:50.410 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.172419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142030.log 2023-12-24T21:18:50.410 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.172419.log.gz 2023-12-24T21:18:50.410 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148726.log 2023-12-24T21:18:50.411 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122996.log.gz 2023-12-24T21:18:50.411 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142030.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142030.log.gz 2023-12-24T21:18:50.411 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137237.log 2023-12-24T21:18:50.411 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150797.log 2023-12-24T21:18:50.411 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148726.log.gz 2023-12-24T21:18:50.411 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162490.log 2023-12-24T21:18:50.411 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137237.log.gz 2023-12-24T21:18:50.411 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152134.log 2023-12-24T21:18:50.411 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150797.log.gz 2023-12-24T21:18:50.412 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161778.log 2023-12-24T21:18:50.412 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162490.log.gz 2023-12-24T21:18:50.412 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152699.log 2023-12-24T21:18:50.412 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152134.log.gz 2023-12-24T21:18:50.412 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169782.log 2023-12-24T21:18:50.412 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161778.log.gz 2023-12-24T21:18:50.412 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154293.log 2023-12-24T21:18:50.412 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152699.log.gz 2023-12-24T21:18:50.412 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.169782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161249.log 0.0% 2023-12-24T21:18:50.413 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.169782.log.gz 2023-12-24T21:18:50.413 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156683.log 2023-12-24T21:18:50.413 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154293.log.gz 2023-12-24T21:18:50.413 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157666.log 2023-12-24T21:18:50.413 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161249.log.gz 2023-12-24T21:18:50.413 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156683.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156683.log.gz 2023-12-24T21:18:50.413 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121542.log 2023-12-24T21:18:50.413 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152466.log 2023-12-24T21:18:50.414 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157666.log.gz 2023-12-24T21:18:50.414 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135709.log 2023-12-24T21:18:50.414 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121542.log.gz 2023-12-24T21:18:50.414 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131391.log 2023-12-24T21:18:50.414 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152466.log.gz 2023-12-24T21:18:50.414 INFO:teuthology.orchestra.run.smithi087.stderr:gzip /var/log/ceph/ceph-client.admin.135709.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.136285.log 2023-12-24T21:18:50.414 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135709.log.gz 2023-12-24T21:18:50.414 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123275.log 2023-12-24T21:18:50.414 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131391.log.gz 2023-12-24T21:18:50.415 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136285.log: 0.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.136285.log.gz--verbose 2023-12-24T21:18:50.415 INFO:teuthology.orchestra.run.smithi087.stderr:-- /var/log/ceph/ceph-client.admin.131799.log 2023-12-24T21:18:50.415 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125723.log 2023-12-24T21:18:50.415 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123275.log.gz 2023-12-24T21:18:50.415 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158372.log 2023-12-24T21:18:50.415 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131799.log.gz 2023-12-24T21:18:50.415 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145647.log 2023-12-24T21:18:50.415 INFO:teuthology.orchestra.run.smithi087.stderr: 23.1% -- replaced with /var/log/ceph/ceph-client.admin.125723.log.gz 2023-12-24T21:18:50.415 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125510.log 2023-12-24T21:18:50.416 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158372.log.gz 2023-12-24T21:18:50.416 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167932.log 2023-12-24T21:18:50.416 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145647.log.gz 2023-12-24T21:18:50.416 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125510.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125510.log.gz 2023-12-24T21:18:50.416 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132333.log 2023-12-24T21:18:50.416 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.167932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161227.log 2023-12-24T21:18:50.416 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167932.log.gz 2023-12-24T21:18:50.416 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152620.log 2023-12-24T21:18:50.416 INFO:teuthology.orchestra.run.smithi087.stderr: 23.4% -- replaced with /var/log/ceph/ceph-client.admin.132333.log.gz 2023-12-24T21:18:50.417 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161227.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161227.log.gz 2023-12-24T21:18:50.417 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144478.log 2023-12-24T21:18:50.417 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158332.log 2023-12-24T21:18:50.417 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152620.log.gz 2023-12-24T21:18:50.417 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122216.log 2023-12-24T21:18:50.417 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144478.log.gz 2023-12-24T21:18:50.417 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147858.log 2023-12-24T21:18:50.417 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158332.log.gz 2023-12-24T21:18:50.417 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160133.log 2023-12-24T21:18:50.418 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.122216.log.gz 2023-12-24T21:18:50.418 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155782.log 2023-12-24T21:18:50.418 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147858.log.gz 2023-12-24T21:18:50.418 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161055.log /var/log/ceph/ceph-client.admin.160133.log: 2023-12-24T21:18:50.418 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160133.log.gz 2023-12-24T21:18:50.418 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158725.log 2023-12-24T21:18:50.418 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155782.log.gz 2023-12-24T21:18:50.418 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132816.log 2023-12-24T21:18:50.419 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161055.log.gz 2023-12-24T21:18:50.419 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130966.log 2023-12-24T21:18:50.419 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158725.log.gz 2023-12-24T21:18:50.419 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159762.log 2023-12-24T21:18:50.419 INFO:teuthology.orchestra.run.smithi087.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.132816.log.gz 2023-12-24T21:18:50.419 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146500.log 2023-12-24T21:18:50.419 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130966.log.gz 2023-12-24T21:18:50.419 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123670.log 2023-12-24T21:18:50.419 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159762.log.gz 2023-12-24T21:18:50.420 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130626.log 2023-12-24T21:18:50.420 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146500.log.gz 2023-12-24T21:18:50.420 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120358.log 2023-12-24T21:18:50.420 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123670.log.gz 2023-12-24T21:18:50.420 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130626.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.130626.log.gz-5 2023-12-24T21:18:50.420 INFO:teuthology.orchestra.run.smithi087.stderr:--verbose -- /var/log/ceph/ceph-client.admin.151025.log 2023-12-24T21:18:50.420 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.120358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149350.log 2023-12-24T21:18:50.420 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120358.log.gz 2023-12-24T21:18:50.420 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133782.log 2023-12-24T21:18:50.421 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151025.log.gz 2023-12-24T21:18:50.421 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133236.log 2023-12-24T21:18:50.421 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149350.log.gz 2023-12-24T21:18:50.421 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146054.log 2023-12-24T21:18:50.421 INFO:teuthology.orchestra.run.smithi087.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.133782.log.gz 2023-12-24T21:18:50.421 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139909.log 2023-12-24T21:18:50.421 INFO:teuthology.orchestra.run.smithi087.stderr: 23.4% -- replaced with /var/log/ceph/ceph-client.admin.133236.log.gz 2023-12-24T21:18:50.421 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133110.log 2023-12-24T21:18:50.421 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146054.log.gz 2023-12-24T21:18:50.422 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147817.log 2023-12-24T21:18:50.422 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139909.log.gz 2023-12-24T21:18:50.422 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131051.log 2023-12-24T21:18:50.422 INFO:teuthology.orchestra.run.smithi087.stderr: 23.4% -- replaced with /var/log/ceph/ceph-client.admin.133110.log.gz 2023-12-24T21:18:50.422 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152228.log 2023-12-24T21:18:50.422 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147817.log.gz 2023-12-24T21:18:50.422 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123189.log 2023-12-24T21:18:50.422 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131051.log.gz 2023-12-24T21:18:50.423 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158115.log 2023-12-24T21:18:50.423 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152228.log.gz 2023-12-24T21:18:50.423 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122039.log 2023-12-24T21:18:50.423 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123189.log.gz 2023-12-24T21:18:50.423 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123146.log 2023-12-24T21:18:50.423 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158115.log.gz 2023-12-24T21:18:50.423 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148640.log 2023-12-24T21:18:50.423 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122039.log.gz 2023-12-24T21:18:50.423 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131119.log 2023-12-24T21:18:50.423 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123146.log.gz 2023-12-24T21:18:50.424 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120409.log 2023-12-24T21:18:50.424 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148640.log.gz 2023-12-24T21:18:50.424 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131119.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-12-24T21:18:50.424 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131119.log.gz 2023-12-24T21:18:50.424 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.120409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135580.log 2023-12-24T21:18:50.424 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120409.log.gz 2023-12-24T21:18:50.424 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142714.log 2023-12-24T21:18:50.424 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121370.log 2023-12-24T21:18:50.425 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135580.log.gz 2023-12-24T21:18:50.425 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126456.log 2023-12-24T21:18:50.425 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142714.log.gz 2023-12-24T21:18:50.425 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121198.log 2023-12-24T21:18:50.425 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121370.log.gz 2023-12-24T21:18:50.425 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126176.log 2023-12-24T21:18:50.425 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126456.log.gz 2023-12-24T21:18:50.425 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136513.log 2023-12-24T21:18:50.425 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121198.log.gz 2023-12-24T21:18:50.426 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124326.log 2023-12-24T21:18:50.426 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126176.log.gz 2023-12-24T21:18:50.426 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171627.log 2023-12-24T21:18:50.426 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136513.log.gz 2023-12-24T21:18:50.426 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128793.log 2023-12-24T21:18:50.426 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124326.log.gz 2023-12-24T21:18:50.426 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.171627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149071.log 2023-12-24T21:18:50.426 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171627.log.gz 2023-12-24T21:18:50.426 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128793.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128793.log.gz 2023-12-24T21:18:50.427 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158352.log 2023-12-24T21:18:50.427 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128493.log 2023-12-24T21:18:50.427 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149071.log.gz 2023-12-24T21:18:50.427 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153992.log 2023-12-24T21:18:50.427 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158352.log.gz 2023-12-24T21:18:50.427 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160346.log 2023-12-24T21:18:50.427 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128493.log.gz 2023-12-24T21:18:50.427 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153992.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153992.log.gz 2023-12-24T21:18:50.427 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139780.log 2023-12-24T21:18:50.428 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168834.log 2023-12-24T21:18:50.428 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160346.log.gz 2023-12-24T21:18:50.428 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139780.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139780.log.gz 2023-12-24T21:18:50.428 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157845.log 2023-12-24T21:18:50.428 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.168834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148554.log 2023-12-24T21:18:50.428 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168834.log.gz 2023-12-24T21:18:50.428 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157845.log: gzip -5 0.0%--verbose -- replaced with /var/log/ceph/ceph-client.admin.157845.log.gz-- 2023-12-24T21:18:50.428 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140016.log 2023-12-24T21:18:50.428 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157808.log 2023-12-24T21:18:50.429 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148554.log.gz 2023-12-24T21:18:50.429 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132122.log 2023-12-24T21:18:50.429 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140016.log.gz 2023-12-24T21:18:50.429 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134846.log 2023-12-24T21:18:50.429 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157808.log.gz 2023-12-24T21:18:50.429 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119917.log 2023-12-24T21:18:50.429 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132122.log.gz 2023-12-24T21:18:50.429 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141664.log 2023-12-24T21:18:50.429 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134846.log.gz 2023-12-24T21:18:50.430 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.119917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131221.log 2023-12-24T21:18:50.430 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119917.log.gz 2023-12-24T21:18:50.430 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156227.log 2023-12-24T21:18:50.430 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141664.log.gz 2023-12-24T21:18:50.430 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157433.log 2023-12-24T21:18:50.430 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131221.log.gz 2023-12-24T21:18:50.430 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122198.log 2023-12-24T21:18:50.430 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156227.log.gz 2023-12-24T21:18:50.430 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125121.log 2023-12-24T21:18:50.431 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157433.log.gz 2023-12-24T21:18:50.431 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123254.log 2023-12-24T21:18:50.431 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122198.log.gz 2023-12-24T21:18:50.431 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134889.log 2023-12-24T21:18:50.431 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125121.log.gz 2023-12-24T21:18:50.431 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123254.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.148877.log 2023-12-24T21:18:50.431 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.123254.log.gz 2023-12-24T21:18:50.431 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136493.log 2023-12-24T21:18:50.431 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134889.log.gz 2023-12-24T21:18:50.432 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155602.log 2023-12-24T21:18:50.432 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148877.log.gz 2023-12-24T21:18:50.432 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155035.log 2023-12-24T21:18:50.432 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136493.log.gz 2023-12-24T21:18:50.432 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155602.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155602.log.gz 2023-12-24T21:18:50.432 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158647.log 2023-12-24T21:18:50.432 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159323.log 2023-12-24T21:18:50.432 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155035.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155035.log.gz 2023-12-24T21:18:50.433 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135644.log 2023-12-24T21:18:50.433 INFO:teuthology.orchestra.run.smithi087.stderr: 55.2% -- replaced with /var/log/ceph/ceph-client.admin.158647.log.gz 2023-12-24T21:18:50.433 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133152.log 2023-12-24T21:18:50.433 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159323.log.gz 2023-12-24T21:18:50.433 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171529.log 2023-12-24T21:18:50.433 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135644.log.gz 2023-12-24T21:18:50.433 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133005.log 2023-12-24T21:18:50.433 INFO:teuthology.orchestra.run.smithi087.stderr: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.133152.log.gz 2023-12-24T21:18:50.433 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.171529.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171529.log.gz 2023-12-24T21:18:50.434 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141557.log 2023-12-24T21:18:50.434 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159656.log 2023-12-24T21:18:50.434 INFO:teuthology.orchestra.run.smithi087.stderr: 23.4% -- replaced with /var/log/ceph/ceph-client.admin.133005.log.gz 2023-12-24T21:18:50.434 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133908.log 2023-12-24T21:18:50.434 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141557.log.gz 2023-12-24T21:18:50.434 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125854.log 2023-12-24T21:18:50.434 INFO:teuthology.orchestra.run.smithi087.stderr: 57.2% -- replaced with /var/log/ceph/ceph-client.admin.159656.log.gz 2023-12-24T21:18:50.434 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132054.log 2023-12-24T21:18:50.435 INFO:teuthology.orchestra.run.smithi087.stderr: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.133908.log.gz 2023-12-24T21:18:50.435 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145410.log 2023-12-24T21:18:50.435 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125854.log.gz 2023-12-24T21:18:50.435 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132054.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132054.log.gz 2023-12-24T21:18:50.435 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154921.log 2023-12-24T21:18:50.435 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146349.log 2023-12-24T21:18:50.435 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145410.log.gz 2023-12-24T21:18:50.435 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154921.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154921.log.gz 2023-12-24T21:18:50.435 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127401.log 2023-12-24T21:18:50.436 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123554.log 2023-12-24T21:18:50.436 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146349.log.gz 2023-12-24T21:18:50.436 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143789.log 2023-12-24T21:18:50.436 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127401.log.gz 2023-12-24T21:18:50.436 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151083.log 2023-12-24T21:18:50.436 INFO:teuthology.orchestra.run.smithi087.stderr: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.123554.log.gz 2023-12-24T21:18:50.436 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150840.log 2023-12-24T21:18:50.436 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143789.log.gz 2023-12-24T21:18:50.436 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151083.log: gzip -5 0.0%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.151083.log.gz/var/log/ceph/ceph-client.admin.128149.log 2023-12-24T21:18:50.437 INFO:teuthology.orchestra.run.smithi087.stderr: 2023-12-24T21:18:50.437 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151983.log 2023-12-24T21:18:50.437 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150840.log.gz 2023-12-24T21:18:50.437 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122084.log 2023-12-24T21:18:50.437 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.128149.log.gz 2023-12-24T21:18:50.437 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151983.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.151983.log.gz-5 2023-12-24T21:18:50.437 INFO:teuthology.orchestra.run.smithi087.stderr:--verbose -- /var/log/ceph/ceph-client.admin.123361.log 2023-12-24T21:18:50.437 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143046.log 2023-12-24T21:18:50.437 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122084.log.gz 2023-12-24T21:18:50.438 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133215.log 2023-12-24T21:18:50.438 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123361.log.gz 2023-12-24T21:18:50.438 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143046.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143046.log.gz 2023-12-24T21:18:50.438 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127231.log 2023-12-24T21:18:50.438 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146457.log 2023-12-24T21:18:50.438 INFO:teuthology.orchestra.run.smithi087.stderr: 23.6% -- replaced with /var/log/ceph/ceph-client.admin.133215.log.gz 2023-12-24T21:18:50.438 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148061.log 2023-12-24T21:18:50.438 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127231.log.gz 2023-12-24T21:18:50.438 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154659.log 2023-12-24T21:18:50.439 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146457.log.gz 2023-12-24T21:18:50.439 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142941.log 2023-12-24T21:18:50.439 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148061.log.gz 2023-12-24T21:18:50.439 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147732.log 2023-12-24T21:18:50.439 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154659.log.gz 2023-12-24T21:18:50.439 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126477.log 2023-12-24T21:18:50.439 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142941.log.gz 2023-12-24T21:18:50.439 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138539.log 2023-12-24T21:18:50.439 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147732.log.gz 2023-12-24T21:18:50.440 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129233.log 2023-12-24T21:18:50.440 INFO:teuthology.orchestra.run.smithi087.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.126477.log.gz 2023-12-24T21:18:50.440 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153019.log 2023-12-24T21:18:50.440 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138539.log.gz 2023-12-24T21:18:50.440 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134524.log 2023-12-24T21:18:50.440 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129233.log.gz 2023-12-24T21:18:50.440 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160475.log 2023-12-24T21:18:50.440 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153019.log.gz 2023-12-24T21:18:50.440 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144903.log 2023-12-24T21:18:50.441 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134524.log.gz 2023-12-24T21:18:50.441 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160776.log 2023-12-24T21:18:50.441 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160475.log.gz 2023-12-24T21:18:50.441 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132105.log 2023-12-24T21:18:50.441 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144903.log.gz 2023-12-24T21:18:50.441 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127486.log 2023-12-24T21:18:50.441 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160776.log.gz 2023-12-24T21:18:50.441 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162662.log 2023-12-24T21:18:50.441 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132105.log.gz 2023-12-24T21:18:50.442 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141492.log 2023-12-24T21:18:50.442 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127486.log.gz 2023-12-24T21:18:50.442 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126908.log 2023-12-24T21:18:50.442 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162662.log.gz 2023-12-24T21:18:50.442 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139930.log 2023-12-24T21:18:50.442 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141492.log.gz 2023-12-24T21:18:50.442 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136596.log 2023-12-24T21:18:50.442 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.126908.log.gz 2023-12-24T21:18:50.442 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137671.log 2023-12-24T21:18:50.442 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139930.log.gz 2023-12-24T21:18:50.443 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121585.log 2023-12-24T21:18:50.443 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136596.log.gz 2023-12-24T21:18:50.443 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128713.log 2023-12-24T21:18:50.443 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137671.log.gz 2023-12-24T21:18:50.443 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134760.log 2023-12-24T21:18:50.443 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121585.log.gz 2023-12-24T21:18:50.443 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141124.log 2023-12-24T21:18:50.443 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128713.log.gz 2023-12-24T21:18:50.444 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132522.log 2023-12-24T21:18:50.444 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134760.log.gz 2023-12-24T21:18:50.444 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156207.log 2023-12-24T21:18:50.444 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141124.log.gz 2023-12-24T21:18:50.444 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150087.log 2023-12-24T21:18:50.444 INFO:teuthology.orchestra.run.smithi087.stderr: 23.9% -- replaced with /var/log/ceph/ceph-client.admin.132522.log.gz 2023-12-24T21:18:50.444 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156207.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156207.log.gz 2023-12-24T21:18:50.444 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145217.log 2023-12-24T21:18:50.444 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150387.log 2023-12-24T21:18:50.445 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150087.log.gz 2023-12-24T21:18:50.445 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145217.log: gzip 0.0%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.145217.log.gz-- 2023-12-24T21:18:50.445 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125592.log 2023-12-24T21:18:50.445 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123809.log 2023-12-24T21:18:50.445 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150387.log.gz 2023-12-24T21:18:50.445 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146801.log 2023-12-24T21:18:50.445 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125592.log.gz 2023-12-24T21:18:50.445 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155803.log 2023-12-24T21:18:50.445 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123809.log.gz 2023-12-24T21:18:50.446 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131680.log 2023-12-24T21:18:50.446 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146801.log.gz 2023-12-24T21:18:50.446 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135558.log 2023-12-24T21:18:50.446 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155803.log.gz 2023-12-24T21:18:50.446 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129893.log 2023-12-24T21:18:50.446 INFO:teuthology.orchestra.run.smithi087.stderr: 28.5% -- replaced with /var/log/ceph/ceph-client.admin.131680.log.gz 2023-12-24T21:18:50.446 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133173.log 2023-12-24T21:18:50.446 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135558.log.gz 2023-12-24T21:18:50.447 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126959.log 2023-12-24T21:18:50.447 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129893.log.gz 2023-12-24T21:18:50.447 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163100.log 2023-12-24T21:18:50.447 INFO:teuthology.orchestra.run.smithi087.stderr: 23.8% -- replaced with /var/log/ceph/ceph-client.admin.133173.log.gz 2023-12-24T21:18:50.447 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143918.log 2023-12-24T21:18:50.447 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.126959.log.gz 2023-12-24T21:18:50.447 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161636.log 2023-12-24T21:18:50.447 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163100.log.gz 2023-12-24T21:18:50.447 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119574.log 2023-12-24T21:18:50.448 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143918.log.gz 2023-12-24T21:18:50.448 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167883.log 2023-12-24T21:18:50.448 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161636.log.gz 2023-12-24T21:18:50.448 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.119574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131782.log 2023-12-24T21:18:50.448 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119574.log.gz 2023-12-24T21:18:50.448 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.167883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136306.log 2023-12-24T21:18:50.448 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167883.log.gz 2023-12-24T21:18:50.448 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124970.log 2023-12-24T21:18:50.448 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131782.log.gz 2023-12-24T21:18:50.449 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136306.log: gzip -5 0.0%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.136306.log.gz/var/log/ceph/ceph-client.admin.148040.log 2023-12-24T21:18:50.449 INFO:teuthology.orchestra.run.smithi087.stderr: 2023-12-24T21:18:50.449 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156607.log 2023-12-24T21:18:50.449 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124970.log.gz 2023-12-24T21:18:50.449 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132627.log 2023-12-24T21:18:50.449 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148040.log.gz 2023-12-24T21:18:50.449 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143446.log 2023-12-24T21:18:50.449 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156607.log.gz 2023-12-24T21:18:50.450 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166194.log 2023-12-24T21:18:50.450 INFO:teuthology.orchestra.run.smithi087.stderr: 23.1% -- replaced with /var/log/ceph/ceph-client.admin.132627.log.gz 2023-12-24T21:18:50.450 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128319.log 2023-12-24T21:18:50.450 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143446.log.gz 2023-12-24T21:18:50.450 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.166194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132963.log 2023-12-24T21:18:50.450 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166194.log.gz 2023-12-24T21:18:50.450 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128200.log 2023-12-24T21:18:50.450 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.128319.log.gz 2023-12-24T21:18:50.450 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168128.log 2023-12-24T21:18:50.451 INFO:teuthology.orchestra.run.smithi087.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.132963.log.gz 2023-12-24T21:18:50.451 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146865.log 2023-12-24T21:18:50.451 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.128200.log.gz 2023-12-24T21:18:50.451 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.168128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166674.log 2023-12-24T21:18:50.451 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168128.log.gz 2023-12-24T21:18:50.451 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152176.log 2023-12-24T21:18:50.451 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146865.log.gz 2023-12-24T21:18:50.451 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.166674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131986.log 2023-12-24T21:18:50.452 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166674.log.gz 2023-12-24T21:18:50.452 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152176.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152176.log.gz 2023-12-24T21:18:50.452 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140037.log 2023-12-24T21:18:50.452 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131986.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131986.log.gz 2023-12-24T21:18:50.452 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123426.log 2023-12-24T21:18:50.452 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161034.log 2023-12-24T21:18:50.452 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140037.log.gz 2023-12-24T21:18:50.452 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148210.log 2023-12-24T21:18:50.452 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123426.log.gz 2023-12-24T21:18:50.452 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150258.log 2023-12-24T21:18:50.453 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161034.log.gz 2023-12-24T21:18:50.453 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148210.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148210.log.gz 2023-12-24T21:18:50.453 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139952.log 2023-12-24T21:18:50.453 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150067.log 2023-12-24T21:18:50.453 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150258.log.gz 2023-12-24T21:18:50.453 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122558.log 2023-12-24T21:18:50.453 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139952.log.gz 2023-12-24T21:18:50.453 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154637.log 2023-12-24T21:18:50.454 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150067.log.gz 2023-12-24T21:18:50.454 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162791.log 2023-12-24T21:18:50.454 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122558.log.gz 2023-12-24T21:18:50.454 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132795.log 2023-12-24T21:18:50.454 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154637.log.gz 2023-12-24T21:18:50.454 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147443.log 2023-12-24T21:18:50.454 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162791.log.gz 2023-12-24T21:18:50.454 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132795.log: 23.5% -- replaced with /var/log/ceph/ceph-client.admin.132795.log.gz 2023-12-24T21:18:50.454 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140127.log 2023-12-24T21:18:50.455 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146978.log 2023-12-24T21:18:50.455 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147443.log.gz 2023-12-24T21:18:50.455 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136348.log 2023-12-24T21:18:50.455 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140127.log.gz 2023-12-24T21:18:50.455 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151123.log 2023-12-24T21:18:50.455 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146978.log.gz 2023-12-24T21:18:50.455 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143067.log 2023-12-24T21:18:50.455 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136348.log.gz 2023-12-24T21:18:50.455 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121026.log 2023-12-24T21:18:50.456 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151123.log.gz 2023-12-24T21:18:50.456 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155706.log 2023-12-24T21:18:50.456 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143067.log.gz 2023-12-24T21:18:50.456 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138221.log 2023-12-24T21:18:50.456 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121026.log.gz 2023-12-24T21:18:50.456 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124240.log 2023-12-24T21:18:50.456 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155706.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155706.log.gz 2023-12-24T21:18:50.456 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158850.log 2023-12-24T21:18:50.456 INFO:teuthology.orchestra.run.smithi087.stderr: 23.8% -- replaced with /var/log/ceph/ceph-client.admin.138221.log.gz 2023-12-24T21:18:50.457 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163270.log 2023-12-24T21:18:50.457 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124240.log.gz 2023-12-24T21:18:50.457 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145238.log 2023-12-24T21:18:50.457 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158850.log.gz 2023-12-24T21:18:50.457 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137607.log 2023-12-24T21:18:50.457 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163270.log.gz 2023-12-24T21:18:50.457 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142073.log 2023-12-24T21:18:50.457 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145238.log.gz 2023-12-24T21:18:50.457 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133971.log 2023-12-24T21:18:50.458 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137607.log.gz 2023-12-24T21:18:50.458 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142073.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142073.log.gz 2023-12-24T21:18:50.458 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130864.log 2023-12-24T21:18:50.458 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149688.log 2023-12-24T21:18:50.458 INFO:teuthology.orchestra.run.smithi087.stderr: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.133971.log.gz 2023-12-24T21:18:50.458 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139973.log 2023-12-24T21:18:50.458 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130864.log.gz 2023-12-24T21:18:50.458 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141232.log 2023-12-24T21:18:50.459 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149688.log.gz 2023-12-24T21:18:50.459 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172068.log 2023-12-24T21:18:50.459 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139973.log.gz 2023-12-24T21:18:50.459 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127418.log 2023-12-24T21:18:50.459 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141232.log.gz 2023-12-24T21:18:50.459 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.172068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138242.log 2023-12-24T21:18:50.459 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.172068.log.gz 2023-12-24T21:18:50.459 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150733.log 2023-12-24T21:18:50.459 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127418.log.gz 2023-12-24T21:18:50.460 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137920.log 2023-12-24T21:18:50.460 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138242.log.gz 2023-12-24T21:18:50.460 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140753.log 2023-12-24T21:18:50.460 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150733.log.gz 2023-12-24T21:18:50.460 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140966.log 2023-12-24T21:18:50.460 INFO:teuthology.orchestra.run.smithi087.stderr: 22.3% -- replaced with /var/log/ceph/ceph-client.admin.137920.log.gz 2023-12-24T21:18:50.460 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133341.log 2023-12-24T21:18:50.460 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140753.log.gz 2023-12-24T21:18:50.460 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138283.log 2023-12-24T21:18:50.461 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140966.log.gz 2023-12-24T21:18:50.461 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162899.log 2023-12-24T21:18:50.461 INFO:teuthology.orchestra.run.smithi087.stderr: 23.5% -- replaced with /var/log/ceph/ceph-client.admin.133341.log.gz 2023-12-24T21:18:50.461 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164170.log 2023-12-24T21:18:50.461 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.138283.log.gz 2023-12-24T21:18:50.461 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154942.log 2023-12-24T21:18:50.461 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162899.log.gz 2023-12-24T21:18:50.461 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.164170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123831.log 2023-12-24T21:18:50.462 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164170.log.gz 2023-12-24T21:18:50.462 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137381.log 2023-12-24T21:18:50.462 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154942.log.gz 2023-12-24T21:18:50.462 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137257.log 2023-12-24T21:18:50.462 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123831.log.gz 2023-12-24T21:18:50.462 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142247.log 2023-12-24T21:18:50.462 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137381.log.gz 2023-12-24T21:18:50.462 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162006.log 2023-12-24T21:18:50.462 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137257.log.gz 2023-12-24T21:18:50.463 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159194.log 2023-12-24T21:18:50.463 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142247.log.gz 2023-12-24T21:18:50.463 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163998.log 2023-12-24T21:18:50.463 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162006.log.gz 2023-12-24T21:18:50.463 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135018.log 2023-12-24T21:18:50.463 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159194.log.gz 2023-12-24T21:18:50.463 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140149.log 2023-12-24T21:18:50.463 INFO:teuthology.orchestra.run.smithi087.stderr: 29.6% -- replaced with /var/log/ceph/ceph-client.admin.163998.log.gz 2023-12-24T21:18:50.463 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114408.log 2023-12-24T21:18:50.464 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135018.log.gz 2023-12-24T21:18:50.464 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124755.log 2023-12-24T21:18:50.464 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140149.log.gz 2023-12-24T21:18:50.464 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.114408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154809.log 2023-12-24T21:18:50.464 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114408.log.gz 2023-12-24T21:18:50.464 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171676.log 2023-12-24T21:18:50.464 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124755.log.gz 2023-12-24T21:18:50.464 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170974.log 2023-12-24T21:18:50.464 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154809.log.gz 2023-12-24T21:18:50.465 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.171676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158192.log 2023-12-24T21:18:50.465 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171676.log.gz 2023-12-24T21:18:50.465 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.170974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136472.log 2023-12-24T21:18:50.465 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170974.log.gz 2023-12-24T21:18:50.465 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129993.log 2023-12-24T21:18:50.465 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158192.log.gz 2023-12-24T21:18:50.465 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124262.log 2023-12-24T21:18:50.465 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136472.log.gz 2023-12-24T21:18:50.466 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162291.log 2023-12-24T21:18:50.466 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129993.log.gz 2023-12-24T21:18:50.466 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126806.log 2023-12-24T21:18:50.466 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124262.log.gz 2023-12-24T21:18:50.466 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135494.log 2023-12-24T21:18:50.466 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162291.log.gz 2023-12-24T21:18:50.466 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150604.log 2023-12-24T21:18:50.466 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.126806.log.gz 2023-12-24T21:18:50.466 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135494.log: gzip -5 0.0%--verbose -- /var/log/ceph/ceph-client.admin.125207.log -- replaced with /var/log/ceph/ceph-client.admin.135494.log.gz 2023-12-24T21:18:50.466 INFO:teuthology.orchestra.run.smithi087.stderr: 2023-12-24T21:18:50.467 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150604.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.150604.log.gz-5 2023-12-24T21:18:50.467 INFO:teuthology.orchestra.run.smithi087.stderr:--verbose -- /var/log/ceph/ceph-client.admin.125835.log 2023-12-24T21:18:50.467 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125799.log 2023-12-24T21:18:50.467 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125207.log.gz 2023-12-24T21:18:50.467 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120703.log 2023-12-24T21:18:50.467 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125835.log.gz 2023-12-24T21:18:50.467 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170223.log 2023-12-24T21:18:50.467 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125799.log.gz 2023-12-24T21:18:50.468 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.120703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127469.log 2023-12-24T21:18:50.468 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120703.log.gz 2023-12-24T21:18:50.468 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.170223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169831.log 2023-12-24T21:18:50.468 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170223.log.gz 2023-12-24T21:18:50.468 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158019.log 2023-12-24T21:18:50.468 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127469.log.gz 2023-12-24T21:18:50.468 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.169831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160539.log 2023-12-24T21:18:50.468 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169831.log.gz 2023-12-24T21:18:50.468 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158019.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.158019.log.gz-5 2023-12-24T21:18:50.469 INFO:teuthology.orchestra.run.smithi087.stderr:--verbose -- /var/log/ceph/ceph-client.admin.139758.log 2023-12-24T21:18:50.469 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141141.log 2023-12-24T21:18:50.469 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160539.log.gz 2023-12-24T21:18:50.469 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155393.log 2023-12-24T21:18:50.469 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139758.log.gz 2023-12-24T21:18:50.469 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125626.log 2023-12-24T21:18:50.469 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141141.log.gz 2023-12-24T21:18:50.469 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138921.log 2023-12-24T21:18:50.469 INFO:teuthology.orchestra.run.smithi087.stderr: 53.5% -- replaced with /var/log/ceph/ceph-client.admin.155393.log.gz 2023-12-24T21:18:50.470 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149646.log 2023-12-24T21:18:50.470 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125626.log.gz 2023-12-24T21:18:50.470 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127741.log 2023-12-24T21:18:50.470 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138921.log.gz 2023-12-24T21:18:50.470 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162813.log 2023-12-24T21:18:50.470 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149646.log.gz 2023-12-24T21:18:50.470 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151553.log 2023-12-24T21:18:50.470 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127741.log.gz 2023-12-24T21:18:50.470 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138958.log 2023-12-24T21:18:50.470 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162813.log.gz 2023-12-24T21:18:50.471 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154315.log 2023-12-24T21:18:50.471 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151553.log.gz 2023-12-24T21:18:50.471 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150236.log 2023-12-24T21:18:50.471 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138958.log.gz 2023-12-24T21:18:50.471 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157962.log 2023-12-24T21:18:50.471 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154315.log.gz 2023-12-24T21:18:50.471 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161679.log 2023-12-24T21:18:50.471 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150236.log.gz 2023-12-24T21:18:50.471 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164192.log 2023-12-24T21:18:50.472 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157962.log.gz 2023-12-24T21:18:50.472 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161012.log 2023-12-24T21:18:50.472 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161679.log.gz 2023-12-24T21:18:50.472 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.164192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163611.log 2023-12-24T21:18:50.472 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164192.log.gz 2023-12-24T21:18:50.472 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124476.log 2023-12-24T21:18:50.472 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161012.log.gz 2023-12-24T21:18:50.472 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125421.log 2023-12-24T21:18:50.472 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163611.log.gz 2023-12-24T21:18:50.473 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132501.log 2023-12-24T21:18:50.473 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124476.log.gz 2023-12-24T21:18:50.473 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124305.log 2023-12-24T21:18:50.473 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.125421.log.gz 2023-12-24T21:18:50.473 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123533.log 2023-12-24T21:18:50.473 INFO:teuthology.orchestra.run.smithi087.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.132501.log.gz 2023-12-24T21:18:50.473 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169880.log 2023-12-24T21:18:50.473 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124305.log.gz 2023-12-24T21:18:50.474 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131068.log 2023-12-24T21:18:50.474 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123533.log.gz 2023-12-24T21:18:50.474 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.169880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163676.log 2023-12-24T21:18:50.474 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169880.log.gz 2023-12-24T21:18:50.474 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141372.log 2023-12-24T21:18:50.474 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131068.log.gz 2023-12-24T21:18:50.474 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139887.log 2023-12-24T21:18:50.474 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163676.log.gz 2023-12-24T21:18:50.474 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128473.log 2023-12-24T21:18:50.475 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141372.log.gz 2023-12-24T21:18:50.475 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160862.log 2023-12-24T21:18:50.475 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139887.log.gz 2023-12-24T21:18:50.475 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131578.log 2023-12-24T21:18:50.475 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128473.log.gz 2023-12-24T21:18:50.475 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124777.log 2023-12-24T21:18:50.475 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160862.log.gz 2023-12-24T21:18:50.475 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123017.log 2023-12-24T21:18:50.476 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131578.log.gz 2023-12-24T21:18:50.476 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150883.log 2023-12-24T21:18:50.476 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124777.log.gz 2023-12-24T21:18:50.476 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153842.log 2023-12-24T21:18:50.476 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123017.log.gz 2023-12-24T21:18:50.476 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125250.log 2023-12-24T21:18:50.476 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150883.log.gz 2023-12-24T21:18:50.476 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157646.log 2023-12-24T21:18:50.476 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153842.log.gz 2023-12-24T21:18:50.477 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145561.log 2023-12-24T21:18:50.477 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125250.log.gz 2023-12-24T21:18:50.477 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157646.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157646.log.gz 2023-12-24T21:18:50.477 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132165.log 2023-12-24T21:18:50.477 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151005.log 2023-12-24T21:18:50.477 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145561.log.gz 2023-12-24T21:18:50.477 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134309.log 2023-12-24T21:18:50.477 INFO:teuthology.orchestra.run.smithi087.stderr: 23.1% -- replaced with /var/log/ceph/ceph-client.admin.132165.log.gz 2023-12-24T21:18:50.478 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162770.log 2023-12-24T21:18:50.478 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151005.log.gz 2023-12-24T21:18:50.478 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149200.log 2023-12-24T21:18:50.478 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134309.log.gz 2023-12-24T21:18:50.478 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162533.log 2023-12-24T21:18:50.478 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162770.log.gz 2023-12-24T21:18:50.478 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125702.log 2023-12-24T21:18:50.478 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149200.log.gz 2023-12-24T21:18:50.478 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133068.log 2023-12-24T21:18:50.479 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162533.log.gz 2023-12-24T21:18:50.479 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122368.log 2023-12-24T21:18:50.479 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125702.log.gz 2023-12-24T21:18:50.479 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121606.log 2023-12-24T21:18:50.479 INFO:teuthology.orchestra.run.smithi087.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.133068.log.gz 2023-12-24T21:18:50.479 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154272.log 2023-12-24T21:18:50.479 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.122368.log.gz 2023-12-24T21:18:50.479 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167083.log 2023-12-24T21:18:50.479 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121606.log.gz 2023-12-24T21:18:50.480 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139543.log 2023-12-24T21:18:50.480 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154272.log.gz 2023-12-24T21:18:50.480 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.167083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144413.log 2023-12-24T21:18:50.480 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167083.log.gz 2023-12-24T21:18:50.480 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149961.log 2023-12-24T21:18:50.480 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139543.log.gz 2023-12-24T21:18:50.480 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163028.log 2023-12-24T21:18:50.480 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144413.log.gz 2023-12-24T21:18:50.480 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133887.log 2023-12-24T21:18:50.481 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149961.log.gz 2023-12-24T21:18:50.481 INFO:teuthology.orchestra.run.smithi087.stderr:gzip /var/log/ceph/ceph-client.admin.163028.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.150473.log 2023-12-24T21:18:50.481 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163028.log.gz 2023-12-24T21:18:50.481 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132417.log 2023-12-24T21:18:50.481 INFO:teuthology.orchestra.run.smithi087.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.133887.log.gz 2023-12-24T21:18:50.481 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149006.log 2023-12-24T21:18:50.481 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150473.log.gz 2023-12-24T21:18:50.481 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142693.log 2023-12-24T21:18:50.482 INFO:teuthology.orchestra.run.smithi087.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.132417.log.gz 2023-12-24T21:18:50.482 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149006.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149006.log.gz 2023-12-24T21:18:50.482 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148019.log 2023-12-24T21:18:50.482 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154100.log 2023-12-24T21:18:50.482 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142693.log.gz 2023-12-24T21:18:50.482 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134481.log 2023-12-24T21:18:50.482 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148019.log.gz 2023-12-24T21:18:50.482 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144263.log 2023-12-24T21:18:50.482 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154100.log.gz 2023-12-24T21:18:50.483 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153519.log 2023-12-24T21:18:50.483 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134481.log.gz 2023-12-24T21:18:50.483 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155294.log 2023-12-24T21:18:50.483 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144263.log.gz 2023-12-24T21:18:50.483 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134739.log 2023-12-24T21:18:50.483 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153519.log.gz 2023-12-24T21:18:50.483 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171072.log 2023-12-24T21:18:50.483 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155294.log.gz 2023-12-24T21:18:50.484 INFO:teuthology.orchestra.run.smithi087.stderr:gzip /var/log/ceph/ceph-client.admin.134739.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.149393.log 2023-12-24T21:18:50.484 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134739.log.gz 2023-12-24T21:18:50.484 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.171072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169422.log 2023-12-24T21:18:50.484 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171072.log.gz 2023-12-24T21:18:50.484 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164149.log 2023-12-24T21:18:50.484 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149393.log.gz 2023-12-24T21:18:50.484 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.169422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161738.log 2023-12-24T21:18:50.484 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169422.log.gz 2023-12-24T21:18:50.484 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.164149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142796.log 2023-12-24T21:18:50.485 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164149.log.gz 2023-12-24T21:18:50.485 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172019.log 2023-12-24T21:18:50.485 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161738.log.gz 2023-12-24T21:18:50.485 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141686.log 2023-12-24T21:18:50.485 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142796.log.gz 2023-12-24T21:18:50.485 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.172019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162428.log 2023-12-24T21:18:50.485 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.172019.log.gz 2023-12-24T21:18:50.485 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159676.log 2023-12-24T21:18:50.485 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141686.log.gz 2023-12-24T21:18:50.486 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169030.log 2023-12-24T21:18:50.486 INFO:teuthology.orchestra.run.smithi087.stderr: 54.1% -- replaced with /var/log/ceph/ceph-client.admin.162428.log.gz 2023-12-24T21:18:50.486 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120260.log 2023-12-24T21:18:50.486 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159676.log.gz 2023-12-24T21:18:50.486 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.169030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138320.log 2023-12-24T21:18:50.486 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169030.log.gz 2023-12-24T21:18:50.486 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.120260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114455.log 2023-12-24T21:18:50.486 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120260.log.gz 2023-12-24T21:18:50.487 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150988.log 2023-12-24T21:18:50.487 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138320.log.gz 2023-12-24T21:18:50.487 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.114455.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-24T21:18:50.487 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114455.log.gz 2023-12-24T21:18:50.487 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170321.log 2023-12-24T21:18:50.487 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150988.log.gz 2023-12-24T21:18:50.487 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145367.log 2023-12-24T21:18:50.487 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.170321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150193.log 2023-12-24T21:18:50.487 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170321.log.gz 2023-12-24T21:18:50.488 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122160.log 2023-12-24T21:18:50.488 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145367.log.gz 2023-12-24T21:18:50.488 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152155.log 2023-12-24T21:18:50.488 INFO:teuthology.orchestra.run.smithi087.stderr: 88.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-24T21:18:50.488 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150193.log.gz 2023-12-24T21:18:50.488 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127588.log 2023-12-24T21:18:50.488 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122160.log.gz 2023-12-24T21:18:50.488 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132732.log 2023-12-24T21:18:50.488 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152155.log.gz 2023-12-24T21:18:50.489 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131731.log 2023-12-24T21:18:50.489 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127588.log.gz 2023-12-24T21:18:50.489 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132732.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.109559.log 2023-12-24T21:18:50.489 INFO:teuthology.orchestra.run.smithi087.stderr: 23.8% -- replaced with /var/log/ceph/ceph-client.admin.132732.log.gz 2023-12-24T21:18:50.489 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136995.log 2023-12-24T21:18:50.489 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131731.log.gz 2023-12-24T21:18:50.489 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph.tmp-client.admin.109559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132037.log 2023-12-24T21:18:50.489 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.109559.log.gz 2023-12-24T21:18:50.489 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163419.log 2023-12-24T21:18:50.490 INFO:teuthology.orchestra.run.smithi087.stderr: 56.1% -- replaced with /var/log/ceph/ceph-client.admin.136995.log.gz 2023-12-24T21:18:50.490 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140516.log 2023-12-24T21:18:50.490 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132037.log.gz 2023-12-24T21:18:50.490 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151424.log 2023-12-24T21:18:50.490 INFO:teuthology.orchestra.run.smithi087.stderr: 53.3% -- replaced with /var/log/ceph/ceph-client.admin.163419.log.gz 2023-12-24T21:18:50.490 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133425.log 0.0% 2023-12-24T21:18:50.490 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.140516.log.gz 2023-12-24T21:18:50.490 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162920.log 2023-12-24T21:18:50.491 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151424.log.gz 2023-12-24T21:18:50.491 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157355.log 2023-12-24T21:18:50.491 INFO:teuthology.orchestra.run.smithi087.stderr: 21.2% -- replaced with /var/log/ceph/ceph-client.admin.133425.log.gz 2023-12-24T21:18:50.491 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162920.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.162920.log.gz-5 2023-12-24T21:18:50.491 INFO:teuthology.orchestra.run.smithi087.stderr:--verbose -- /var/log/ceph/ceph-client.admin.131357.log 2023-12-24T21:18:50.491 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157998.log 2023-12-24T21:18:50.491 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157355.log.gz 2023-12-24T21:18:50.491 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170125.log 2023-12-24T21:18:50.491 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131357.log.gz 2023-12-24T21:18:50.492 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135537.log 2023-12-24T21:18:50.492 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157998.log.gz 2023-12-24T21:18:50.492 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.170125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145091.log 2023-12-24T21:18:50.492 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170125.log.gz 2023-12-24T21:18:50.492 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143467.log 2023-12-24T21:18:50.492 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135537.log.gz 2023-12-24T21:18:50.492 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146328.log 2023-12-24T21:18:50.492 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145091.log.gz 2023-12-24T21:18:50.493 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129173.log 2023-12-24T21:18:50.493 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143467.log.gz 2023-12-24T21:18:50.493 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127367.log 2023-12-24T21:18:50.493 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146328.log.gz 2023-12-24T21:18:50.493 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136654.log 2023-12-24T21:18:50.493 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129173.log.gz 2023-12-24T21:18:50.493 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151747.log 2023-12-24T21:18:50.493 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127367.log.gz 2023-12-24T21:18:50.493 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156287.log 2023-12-24T21:18:50.494 INFO:teuthology.orchestra.run.smithi087.stderr: 23.1% -- replaced with /var/log/ceph/ceph-client.admin.136654.log.gz 2023-12-24T21:18:50.494 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141987.log 2023-12-24T21:18:50.494 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151747.log.gz 2023-12-24T21:18:50.494 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136251.log 2023-12-24T21:18:50.494 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156287.log.gz 2023-12-24T21:18:50.494 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154078.log 2023-12-24T21:18:50.494 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141987.log.gz 2023-12-24T21:18:50.494 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137754.log 2023-12-24T21:18:50.494 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136251.log.gz 2023-12-24T21:18:50.495 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128064.log 2023-12-24T21:18:50.495 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154078.log.gz 2023-12-24T21:18:50.495 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127299.log 2023-12-24T21:18:50.495 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137754.log.gz 2023-12-24T21:18:50.495 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128773.log 2023-12-24T21:18:50.495 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.128064.log.gz 2023-12-24T21:18:50.495 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133194.log 2023-12-24T21:18:50.495 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127299.log.gz 2023-12-24T21:18:50.496 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145453.log 2023-12-24T21:18:50.496 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128773.log.gz 2023-12-24T21:18:50.496 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152386.log 2023-12-24T21:18:50.496 INFO:teuthology.orchestra.run.smithi087.stderr: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.133194.log.gz 2023-12-24T21:18:50.496 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128047.log 2023-12-24T21:18:50.496 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145453.log.gz 2023-12-24T21:18:50.496 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121456.log 2023-12-24T21:18:50.496 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152386.log.gz 2023-12-24T21:18:50.496 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143553.log 2023-12-24T21:18:50.496 INFO:teuthology.orchestra.run.smithi087.stderr: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.128047.log.gz 2023-12-24T21:18:50.497 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144392.log 2023-12-24T21:18:50.497 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121456.log.gz 2023-12-24T21:18:50.497 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151143.log 2023-12-24T21:18:50.497 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143553.log.gz 2023-12-24T21:18:50.497 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122781.log 2023-12-24T21:18:50.497 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144392.log.gz 2023-12-24T21:18:50.497 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140407.log 2023-12-24T21:18:50.497 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151143.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151143.log.gz 2023-12-24T21:18:50.498 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159697.log 2023-12-24T21:18:50.498 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122781.log.gz 2023-12-24T21:18:50.498 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156130.log 2023-12-24T21:18:50.498 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140407.log.gz 2023-12-24T21:18:50.498 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136716.log 2023-12-24T21:18:50.498 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159697.log.gz 2023-12-24T21:18:50.498 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128953.log 2023-12-24T21:18:50.498 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156130.log.gz 2023-12-24T21:18:50.498 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132921.log 2023-12-24T21:18:50.499 INFO:teuthology.orchestra.run.smithi087.stderr: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.136716.log.gz 2023-12-24T21:18:50.499 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128953.log: gzip -5 --verbose 0.0%-- /var/log/ceph/ceph-client.admin.131527.log -- replaced with /var/log/ceph/ceph-client.admin.128953.log.gz 2023-12-24T21:18:50.499 INFO:teuthology.orchestra.run.smithi087.stderr: 2023-12-24T21:18:50.499 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128013.log 2023-12-24T21:18:50.499 INFO:teuthology.orchestra.run.smithi087.stderr: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.132921.log.gz 2023-12-24T21:18:50.499 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168981.log 2023-12-24T21:18:50.499 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131527.log.gz 2023-12-24T21:18:50.499 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158571.log 2023-12-24T21:18:50.499 INFO:teuthology.orchestra.run.smithi087.stderr: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.128013.log.gz 2023-12-24T21:18:50.500 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.168981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159130.log 2023-12-24T21:18:50.500 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168981.log.gz 2023-12-24T21:18:50.500 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134395.log 2023-12-24T21:18:50.500 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158571.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158571.log.gz 2023-12-24T21:18:50.500 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131969.log 2023-12-24T21:18:50.500 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159130.log.gz 2023-12-24T21:18:50.500 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158534.log 2023-12-24T21:18:50.500 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134395.log.gz 2023-12-24T21:18:50.500 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143188.log 2023-12-24T21:18:50.501 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131969.log.gz 2023-12-24T21:18:50.501 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158534.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158534.log.gz 2023-12-24T21:18:50.501 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148274.log 2023-12-24T21:18:50.501 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132669.log 2023-12-24T21:18:50.501 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143188.log.gz 2023-12-24T21:18:50.501 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156644.log 2023-12-24T21:18:50.501 INFO:teuthology.orchestra.run.smithi087.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.148274.log.gz 2023-12-24T21:18:50.501 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160840.log 2023-12-24T21:18:50.502 INFO:teuthology.orchestra.run.smithi087.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.132669.log.gz 2023-12-24T21:18:50.502 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143961.log 2023-12-24T21:18:50.502 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156644.log.gz 2023-12-24T21:18:50.502 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125455.log 2023-12-24T21:18:50.502 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160840.log.gz 2023-12-24T21:18:50.502 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140428.log 2023-12-24T21:18:50.502 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143961.log.gz 2023-12-24T21:18:50.502 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155453.log 2023-12-24T21:18:50.503 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125455.log.gz 2023-12-24T21:18:50.503 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164253.log 2023-12-24T21:18:50.503 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140428.log.gz 2023-12-24T21:18:50.503 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161120.log 2023-12-24T21:18:50.503 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155453.log.gz 2023-12-24T21:18:50.503 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.164253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148618.log 2023-12-24T21:18:50.503 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164253.log.gz 2023-12-24T21:18:50.503 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126047.log 2023-12-24T21:18:50.503 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161120.log.gz 2023-12-24T21:18:50.504 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129533.log 2023-12-24T21:18:50.504 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148618.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148618.log.gz 2023-12-24T21:18:50.504 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134717.log 2023-12-24T21:18:50.504 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126047.log.gz 2023-12-24T21:18:50.504 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122312.log 2023-12-24T21:18:50.504 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129533.log.gz 2023-12-24T21:18:50.504 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156956.log 2023-12-24T21:18:50.504 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134717.log.gz 2023-12-24T21:18:50.504 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120117.log 2023-12-24T21:18:50.505 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122312.log.gz 2023-12-24T21:18:50.505 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144967.log 2023-12-24T21:18:50.505 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156956.log.gz 2023-12-24T21:18:50.505 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.120117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169177.log 2023-12-24T21:18:50.505 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120117.log.gz 2023-12-24T21:18:50.505 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138718.log 2023-12-24T21:18:50.505 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144967.log.gz 2023-12-24T21:18:50.505 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.169177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127860.log 2023-12-24T21:18:50.505 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169177.log.gz 2023-12-24T21:18:50.506 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137484.log 2023-12-24T21:18:50.506 INFO:teuthology.orchestra.run.smithi087.stderr: 22.3% -- replaced with /var/log/ceph/ceph-client.admin.138718.log.gz 2023-12-24T21:18:50.506 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134352.log 2023-12-24T21:18:50.506 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127860.log.gz 2023-12-24T21:18:50.506 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129013.log 2023-12-24T21:18:50.506 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137484.log.gz 2023-12-24T21:18:50.506 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163633.log 2023-12-24T21:18:50.506 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134352.log.gz 2023-12-24T21:18:50.507 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130490.log 2023-12-24T21:18:50.507 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129013.log.gz 2023-12-24T21:18:50.507 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133677.log 2023-12-24T21:18:50.507 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163633.log.gz 2023-12-24T21:18:50.507 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133551.log 2023-12-24T21:18:50.507 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130490.log.gz 2023-12-24T21:18:50.507 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145625.log 2023-12-24T21:18:50.507 INFO:teuthology.orchestra.run.smithi087.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.133677.log.gz 2023-12-24T21:18:50.507 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158979.log 2023-12-24T21:18:50.507 INFO:teuthology.orchestra.run.smithi087.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.133551.log.gz 2023-12-24T21:18:50.508 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150668.log 2023-12-24T21:18:50.508 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145625.log.gz 2023-12-24T21:18:50.508 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148899.log 2023-12-24T21:18:50.508 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158979.log.gz 2023-12-24T21:18:50.508 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157392.log 2023-12-24T21:18:50.508 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150668.log.gz 2023-12-24T21:18:50.508 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146844.log 2023-12-24T21:18:50.508 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148899.log.gz 2023-12-24T21:18:50.509 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139801.log 2023-12-24T21:18:50.509 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157392.log.gz 2023-12-24T21:18:50.509 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146158.log 2023-12-24T21:18:50.509 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146844.log.gz 2023-12-24T21:18:50.509 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130233.log 2023-12-24T21:18:50.509 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139801.log.gz 2023-12-24T21:18:50.509 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137733.log 2023-12-24T21:18:50.509 INFO:teuthology.orchestra.run.smithi087.stderr: 54.5% -- replaced with /var/log/ceph/ceph-client.admin.146158.log.gz 2023-12-24T21:18:50.509 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132753.log 2023-12-24T21:18:50.510 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130233.log.gz 2023-12-24T21:18:50.510 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163912.log 2023-12-24T21:18:50.510 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137733.log.gz 2023-12-24T21:18:50.510 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156509.log 2023-12-24T21:18:50.510 INFO:teuthology.orchestra.run.smithi087.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.132753.log.gz 2023-12-24T21:18:50.510 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155935.log 2023-12-24T21:18:50.510 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163912.log.gz 2023-12-24T21:18:50.510 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122045.log 2023-12-24T21:18:50.510 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156509.log.gz 2023-12-24T21:18:50.511 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132088.log 2023-12-24T21:18:50.511 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155935.log.gz 2023-12-24T21:18:50.511 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135752.log 2023-12-24T21:18:50.511 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122045.log.gz 2023-12-24T21:18:50.511 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133089.log 2023-12-24T21:18:50.511 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132088.log.gz 2023-12-24T21:18:50.511 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149522.log 2023-12-24T21:18:50.511 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135752.log.gz 2023-12-24T21:18:50.511 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133614.log 2023-12-24T21:18:50.512 INFO:teuthology.orchestra.run.smithi087.stderr: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.133089.log.gz 2023-12-24T21:18:50.512 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129393.log 2023-12-24T21:18:50.512 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149522.log.gz 2023-12-24T21:18:50.512 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169577.log 2023-12-24T21:18:50.512 INFO:teuthology.orchestra.run.smithi087.stderr: 23.8% -- replaced with /var/log/ceph/ceph-client.admin.133614.log.gz 2023-12-24T21:18:50.512 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168883.log 2023-12-24T21:18:50.512 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129393.log.gz 2023-12-24T21:18:50.512 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.169577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162130.log 2023-12-24T21:18:50.512 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169577.log.gz 2023-12-24T21:18:50.513 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.168883.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168883.log.gz 2023-12-24T21:18:50.513 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162130.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162130.log.gz 2023-12-24T21:18:50.513 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126562.log 2023-12-24T21:18:50.513 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130213.log 2023-12-24T21:18:50.513 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129773.log 2023-12-24T21:18:50.513 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126562.log.gz 2023-12-24T21:18:50.513 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123751.log 2023-12-24T21:18:50.513 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130213.log.gz 2023-12-24T21:18:50.513 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146998.log 2023-12-24T21:18:50.514 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129773.log.gz 2023-12-24T21:18:50.514 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144693.log 2023-12-24T21:18:50.514 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123751.log.gz 2023-12-24T21:18:50.514 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148790.log 2023-12-24T21:18:50.514 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146998.log.gz 2023-12-24T21:18:50.514 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151103.log 2023-12-24T21:18:50.514 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144693.log.gz 2023-12-24T21:18:50.514 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156327.log 2023-12-24T21:18:50.514 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148790.log.gz 2023-12-24T21:18:50.515 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155353.log 2023-12-24T21:18:50.515 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151103.log.gz 2023-12-24T21:18:50.515 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156327.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.155623.log 2023-12-24T21:18:50.515 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.156327.log.gz 2023-12-24T21:18:50.515 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135211.log 2023-12-24T21:18:50.515 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155353.log.gz 2023-12-24T21:18:50.515 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152979.log 2023-12-24T21:18:50.515 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155623.log.gz 2023-12-24T21:18:50.515 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135211.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135211.log.gz 2023-12-24T21:18:50.516 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129413.log 2023-12-24T21:18:50.516 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154401.log 2023-12-24T21:18:50.516 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152979.log.gz 2023-12-24T21:18:50.516 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140084.log 2023-12-24T21:18:50.516 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129413.log.gz 2023-12-24T21:18:50.516 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150044.log 2023-12-24T21:18:50.516 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154401.log.gz 2023-12-24T21:18:50.516 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133929.log 2023-12-24T21:18:50.516 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140084.log.gz 2023-12-24T21:18:50.517 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153541.log 2023-12-24T21:18:50.517 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150044.log.gz 2023-12-24T21:18:50.517 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151768.log 2023-12-24T21:18:50.517 INFO:teuthology.orchestra.run.smithi087.stderr: 23.1% -- replaced with /var/log/ceph/ceph-client.admin.133929.log.gz 2023-12-24T21:18:50.517 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133803.log 2023-12-24T21:18:50.517 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153541.log.gz 2023-12-24T21:18:50.517 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161485.log 2023-12-24T21:18:50.517 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151768.log.gz 2023-12-24T21:18:50.517 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144714.log 2023-12-24T21:18:50.518 INFO:teuthology.orchestra.run.smithi087.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.133803.log.gz 2023-12-24T21:18:50.518 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129513.log 2023-12-24T21:18:50.518 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161485.log.gz 2023-12-24T21:18:50.518 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144241.log 2023-12-24T21:18:50.518 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144714.log.gz 2023-12-24T21:18:50.518 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133467.log 2023-12-24T21:18:50.518 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129513.log.gz 2023-12-24T21:18:50.518 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147018.log 2023-12-24T21:18:50.518 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144241.log.gz 2023-12-24T21:18:50.519 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130353.log 2023-12-24T21:18:50.519 INFO:teuthology.orchestra.run.smithi087.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.133467.log.gz 2023-12-24T21:18:50.519 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156150.log 2023-12-24T21:18:50.519 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147018.log.gz 2023-12-24T21:18:50.519 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133761.log 2023-12-24T21:18:50.519 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130353.log.gz 2023-12-24T21:18:50.519 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132396.log 2023-12-24T21:18:50.519 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156150.log.gz 2023-12-24T21:18:50.519 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132249.log 2023-12-24T21:18:50.520 INFO:teuthology.orchestra.run.smithi087.stderr: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.133761.log.gz 2023-12-24T21:18:50.520 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133257.log 2023-12-24T21:18:50.520 INFO:teuthology.orchestra.run.smithi087.stderr: 23.1% -- replaced with /var/log/ceph/ceph-client.admin.132396.log.gz 2023-12-24T21:18:50.520 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132249.log: /var/log/ceph/ceph-client.admin.129133.log 2023-12-24T21:18:50.520 INFO:teuthology.orchestra.run.smithi087.stderr: 24.2% -- replaced with /var/log/ceph/ceph-client.admin.132249.log.gz 2023-12-24T21:18:50.520 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141815.log 2023-12-24T21:18:50.520 INFO:teuthology.orchestra.run.smithi087.stderr: 23.4% -- replaced with /var/log/ceph/ceph-client.admin.133257.log.gz 2023-12-24T21:18:50.520 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129133.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129133.log.gz 2023-12-24T21:18:50.521 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132141.log 2023-12-24T21:18:50.521 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125531.log 2023-12-24T21:18:50.521 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141815.log.gz 2023-12-24T21:18:50.521 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124927.log 2023-12-24T21:18:50.521 INFO:teuthology.orchestra.run.smithi087.stderr: 56.4% -- replaced with /var/log/ceph/ceph-client.admin.132141.log.gz 2023-12-24T21:18:50.521 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155856.log 2023-12-24T21:18:50.521 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125531.log.gz 2023-12-24T21:18:50.521 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133593.log 2023-12-24T21:18:50.521 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124927.log.gz 2023-12-24T21:18:50.522 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146930.log 2023-12-24T21:18:50.522 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155856.log.gz 2023-12-24T21:18:50.522 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144306.log 2023-12-24T21:18:50.522 INFO:teuthology.orchestra.run.smithi087.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.133593.log.gz 2023-12-24T21:18:50.522 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140842.log 2023-12-24T21:18:50.522 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146930.log.gz 2023-12-24T21:18:50.522 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138143.log 2023-12-24T21:18:50.522 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144306.log.gz 2023-12-24T21:18:50.522 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164287.log 2023-12-24T21:18:50.523 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140842.log.gz 2023-12-24T21:18:50.523 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127962.log 2023-12-24T21:18:50.523 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.138143.log.gz 2023-12-24T21:18:50.523 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.164287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149774.log 2023-12-24T21:18:50.523 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164287.log.gz 2023-12-24T21:18:50.523 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139629.log 2023-12-24T21:18:50.523 INFO:teuthology.orchestra.run.smithi087.stderr: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.127962.log.gz 2023-12-24T21:18:50.523 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153122.log 2023-12-24T21:18:50.524 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149774.log.gz 2023-12-24T21:18:50.524 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149264.log 2023-12-24T21:18:50.524 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139629.log.gz 2023-12-24T21:18:50.524 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161986.log 2023-12-24T21:18:50.524 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153122.log.gz 2023-12-24T21:18:50.524 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148082.log 2023-12-24T21:18:50.524 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149264.log.gz 2023-12-24T21:18:50.524 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153142.log 2023-12-24T21:18:50.524 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161986.log.gz 2023-12-24T21:18:50.525 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156347.log 2023-12-24T21:18:50.525 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148082.log.gz 2023-12-24T21:18:50.525 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139995.log 2023-12-24T21:18:50.525 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153142.log.gz 2023-12-24T21:18:50.525 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124347.log 2023-12-24T21:18:50.525 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156347.log.gz 2023-12-24T21:18:50.525 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122824.log 2023-12-24T21:18:50.525 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139995.log.gz 2023-12-24T21:18:50.525 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129253.log 2023-12-24T21:18:50.526 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124347.log.gz 2023-12-24T21:18:50.526 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138081.log 2023-12-24T21:18:50.526 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122824.log.gz 2023-12-24T21:18:50.526 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122500.log 2023-12-24T21:18:50.526 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129253.log.gz 2023-12-24T21:18:50.526 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119309.log 2023-12-24T21:18:50.526 INFO:teuthology.orchestra.run.smithi087.stderr: 23.1% -- replaced with /var/log/ceph/ceph-client.admin.138081.log.gz 2023-12-24T21:18:50.526 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126112.log 2023-12-24T21:18:50.526 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122500.log.gz 2023-12-24T21:18:50.527 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.119309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150004.log 2023-12-24T21:18:50.527 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119309.log.gz 2023-12-24T21:18:50.527 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126112.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126112.log.gz 2023-12-24T21:18:50.527 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154680.log 2023-12-24T21:18:50.527 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145070.log 2023-12-24T21:18:50.527 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150004.log.gz 2023-12-24T21:18:50.527 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126133.log 2023-12-24T21:18:50.527 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154680.log.gz 2023-12-24T21:18:50.528 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123875.log 2023-12-24T21:18:50.528 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145070.log.gz 2023-12-24T21:18:50.528 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152005.log 2023-12-24T21:18:50.528 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126133.log.gz 2023-12-24T21:18:50.528 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123875.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123875.log.gzgzip 2023-12-24T21:18:50.528 INFO:teuthology.orchestra.run.smithi087.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.140840.log 2023-12-24T21:18:50.528 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148812.log 2023-12-24T21:18:50.528 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152005.log.gz 2023-12-24T21:18:50.528 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140840.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140840.log.gz 2023-12-24T21:18:50.529 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126789.log 2023-12-24T21:18:50.529 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135451.log 2023-12-24T21:18:50.529 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148812.log.gz 2023-12-24T21:18:50.529 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150494.log 2023-12-24T21:18:50.529 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.126789.log.gz 2023-12-24T21:18:50.529 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132020.log 2023-12-24T21:18:50.529 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135451.log.gz 2023-12-24T21:18:50.529 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150494.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.136876.log -- replaced with /var/log/ceph/ceph-client.admin.150494.log.gz 2023-12-24T21:18:50.530 INFO:teuthology.orchestra.run.smithi087.stderr: 2023-12-24T21:18:50.530 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132020.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132020.log.gz 2023-12-24T21:18:50.530 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128853.log 2023-12-24T21:18:50.530 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138900.log 2023-12-24T21:18:50.530 INFO:teuthology.orchestra.run.smithi087.stderr: 23.4% -- replaced with /var/log/ceph/ceph-client.admin.136876.log.gz 2023-12-24T21:18:50.530 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127639.log 2023-12-24T21:18:50.530 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128853.log.gz 2023-12-24T21:18:50.530 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122406.log 2023-12-24T21:18:50.530 INFO:teuthology.orchestra.run.smithi087.stderr: 23.4% -- replaced with /var/log/ceph/ceph-client.admin.138900.log.gz 2023-12-24T21:18:50.531 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128302.log 2023-12-24T21:18:50.531 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127639.log.gz 2023-12-24T21:18:50.531 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151181.log 2023-12-24T21:18:50.531 INFO:teuthology.orchestra.run.smithi087.stderr: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.122406.log.gz 2023-12-24T21:18:50.531 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171219.log 2023-12-24T21:18:50.531 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.128302.log.gz 2023-12-24T21:18:50.531 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163143.log 2023-12-24T21:18:50.531 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151181.log.gz 2023-12-24T21:18:50.531 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.171219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155173.log 2023-12-24T21:18:50.532 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171219.log.gz 2023-12-24T21:18:50.532 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161313.log 2023-12-24T21:18:50.532 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163143.log.gz 2023-12-24T21:18:50.532 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147422.log 2023-12-24T21:18:50.532 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155173.log.gz 2023-12-24T21:18:50.532 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161313.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.161313.log.gz-5 2023-12-24T21:18:50.532 INFO:teuthology.orchestra.run.smithi087.stderr:--verbose -- /var/log/ceph/ceph-client.admin.143269.log 2023-12-24T21:18:50.532 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125078.log 2023-12-24T21:18:50.533 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147422.log.gz 2023-12-24T21:18:50.533 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143269.log: gzip -5 --verbose 0.0%-- -- replaced with /var/log/ceph/ceph-client.admin.143269.log.gz/var/log/ceph/ceph-client.admin.130333.log 2023-12-24T21:18:50.533 INFO:teuthology.orchestra.run.smithi087.stderr: 2023-12-24T21:18:50.533 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121112.log 2023-12-24T21:18:50.533 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125078.log.gz 2023-12-24T21:18:50.533 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130333.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130333.log.gz 2023-12-24T21:18:50.533 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143087.log 2023-12-24T21:18:50.533 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129733.log 2023-12-24T21:18:50.533 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121112.log.gz 2023-12-24T21:18:50.534 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143087.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143087.log.gz 2023-12-24T21:18:50.534 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131374.log 2023-12-24T21:18:50.534 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139307.log 2023-12-24T21:18:50.534 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129733.log.gz 2023-12-24T21:18:50.534 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131374.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131374.log.gz 2023-12-24T21:18:50.534 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171170.log 2023-12-24T21:18:50.534 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142508.log 2023-12-24T21:18:50.534 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139307.log.gz 2023-12-24T21:18:50.534 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.171170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131323.log 2023-12-24T21:18:50.535 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171170.log.gz 2023-12-24T21:18:50.535 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140645.log 2023-12-24T21:18:50.535 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142508.log.gz 2023-12-24T21:18:50.535 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171317.log 2023-12-24T21:18:50.535 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131323.log.gz 2023-12-24T21:18:50.535 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157274.log 2023-12-24T21:18:50.535 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140645.log.gz 2023-12-24T21:18:50.535 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.171317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158512.log 2023-12-24T21:18:50.535 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171317.log.gz 2023-12-24T21:18:50.536 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123082.log 2023-12-24T21:18:50.536 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157274.log.gz 2023-12-24T21:18:50.536 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163208.log 2023-12-24T21:18:50.536 INFO:teuthology.orchestra.run.smithi087.stderr: 56.6% -- replaced with /var/log/ceph/ceph-client.admin.158512.log.gz 2023-12-24T21:18:50.536 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145281.log 2023-12-24T21:18:50.536 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123082.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123082.log.gz 2023-12-24T21:18:50.536 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145260.log 2023-12-24T21:18:50.536 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163208.log.gz 2023-12-24T21:18:50.536 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159065.log 2023-12-24T21:18:50.537 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145281.log.gz 2023-12-24T21:18:50.537 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129873.log 2023-12-24T21:18:50.537 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145260.log.gz 2023-12-24T21:18:50.537 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144349.log 2023-12-24T21:18:50.537 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159065.log.gz 2023-12-24T21:18:50.537 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144499.log 2023-12-24T21:18:50.537 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129873.log.gz 2023-12-24T21:18:50.537 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151532.log 2023-12-24T21:18:50.537 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144349.log.gz 2023-12-24T21:18:50.538 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149049.log 2023-12-24T21:18:50.538 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144499.log.gz 2023-12-24T21:18:50.538 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147038.log 2023-12-24T21:18:50.538 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151532.log.gz 2023-12-24T21:18:50.538 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149307.log 2023-12-24T21:18:50.538 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149049.log.gz 2023-12-24T21:18:50.538 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147038.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.132459.log -- replaced with /var/log/ceph/ceph-client.admin.147038.log.gz 2023-12-24T21:18:50.538 INFO:teuthology.orchestra.run.smithi087.stderr: 2023-12-24T21:18:50.538 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148489.log 2023-12-24T21:18:50.539 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149307.log.gz 2023-12-24T21:18:50.539 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133404.log 2023-12-24T21:18:50.539 INFO:teuthology.orchestra.run.smithi087.stderr: 23.1% -- replaced with /var/log/ceph/ceph-client.admin.132459.log.gz 2023-12-24T21:18:50.539 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162027.log 2023-12-24T21:18:50.539 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148489.log.gz 2023-12-24T21:18:50.539 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125228.log 2023-12-24T21:18:50.539 INFO:teuthology.orchestra.run.smithi087.stderr: 23.8% -- replaced with /var/log/ceph/ceph-client.admin.133404.log.gz 2023-12-24T21:18:50.539 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144112.log 2023-12-24T21:18:50.539 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162027.log.gz 2023-12-24T21:18:50.540 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168030.log 2023-12-24T21:18:50.540 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125228.log.gz 2023-12-24T21:18:50.540 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147611.log 2023-12-24T21:18:50.540 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144112.log.gz 2023-12-24T21:18:50.540 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.168030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134782.log 2023-12-24T21:18:50.540 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168030.log.gz 2023-12-24T21:18:50.540 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135623.log 2023-12-24T21:18:50.540 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147611.log.gz 2023-12-24T21:18:50.540 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141750.log 2023-12-24T21:18:50.541 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134782.log.gz 2023-12-24T21:18:50.541 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139457.log 2023-12-24T21:18:50.541 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135623.log.gz 2023-12-24T21:18:50.541 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149501.log 2023-12-24T21:18:50.541 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141750.log.gz 2023-12-24T21:18:50.541 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146199.log 2023-12-24T21:18:50.541 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139457.log.gz 2023-12-24T21:18:50.541 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120997.log 2023-12-24T21:18:50.541 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149501.log.gz 2023-12-24T21:18:50.541 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152246.log 2023-12-24T21:18:50.542 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146199.log.gz 2023-12-24T21:18:50.542 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.120997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157295.log 2023-12-24T21:18:50.542 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120997.log.gz 2023-12-24T21:18:50.542 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148296.log 2023-12-24T21:18:50.542 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152246.log.gz 2023-12-24T21:18:50.542 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135097.log 2023-12-24T21:18:50.542 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157295.log.gz 2023-12-24T21:18:50.542 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144945.log 2023-12-24T21:18:50.543 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148296.log.gz 2023-12-24T21:18:50.543 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156247.log 2023-12-24T21:18:50.543 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135097.log.gz 2023-12-24T21:18:50.543 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127503.log 2023-12-24T21:18:50.543 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144945.log.gz 2023-12-24T21:18:50.543 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156836.log 2023-12-24T21:18:50.543 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156247.log.gz 2023-12-24T21:18:50.543 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121241.log 2023-12-24T21:18:50.543 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127503.log.gz 2023-12-24T21:18:50.544 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172321.log 2023-12-24T21:18:50.544 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156836.log.gz 2023-12-24T21:18:50.544 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121241.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-12-24T21:18:50.544 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121241.log.gz 2023-12-24T21:18:50.544 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.172321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151596.log 2023-12-24T21:18:50.544 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.172321.log.gz 2023-12-24T21:18:50.544 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163049.log 2023-12-24T21:18:50.544 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151596.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151596.log.gz 2023-12-24T21:18:50.544 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146392.log 2023-12-24T21:18:50.544 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140321.log 2023-12-24T21:18:50.545 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163049.log.gz 2023-12-24T21:18:50.545 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153971.log 2023-12-24T21:18:50.545 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146392.log.gz 2023-12-24T21:18:50.545 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137035.log 2023-12-24T21:18:50.545 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140321.log.gz 2023-12-24T21:18:50.545 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146887.log 2023-12-24T21:18:50.545 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153971.log.gz 2023-12-24T21:18:50.545 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167622.log 2023-12-24T21:18:50.545 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137035.log.gz 2023-12-24T21:18:50.546 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129973.log 2023-12-24T21:18:50.546 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146887.log.gz 2023-12-24T21:18:50.546 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.167622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131697.log 2023-12-24T21:18:50.546 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167622.log.gz 2023-12-24T21:18:50.546 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129973.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129973.log.gz 2023-12-24T21:18:50.546 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133320.log 2023-12-24T21:18:50.546 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137175.log 2023-12-24T21:18:50.546 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131697.log.gz 2023-12-24T21:18:50.546 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143532.log 2023-12-24T21:18:50.547 INFO:teuthology.orchestra.run.smithi087.stderr: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.133320.log.gz 2023-12-24T21:18:50.547 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161571.log 2023-12-24T21:18:50.547 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137175.log.gz 2023-12-24T21:18:50.547 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159431.log 2023-12-24T21:18:50.547 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143532.log.gz 2023-12-24T21:18:50.547 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131017.log 2023-12-24T21:18:50.547 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161571.log.gz 2023-12-24T21:18:50.547 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141471.log 2023-12-24T21:18:50.547 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159431.log.gz 2023-12-24T21:18:50.548 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145604.log 2023-12-24T21:18:50.548 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131017.log.gz 2023-12-24T21:18:50.548 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129933.log 2023-12-24T21:18:50.548 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141471.log.gz 2023-12-24T21:18:50.548 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128268.log 2023-12-24T21:18:50.548 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145604.log.gz 2023-12-24T21:18:50.548 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160281.log 2023-12-24T21:18:50.548 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129933.log.gz 2023-12-24T21:18:50.548 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152935.log 2023-12-24T21:18:50.549 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.128268.log.gz 2023-12-24T21:18:50.549 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125491.log 2023-12-24T21:18:50.549 INFO:teuthology.orchestra.run.smithi087.stderr: 53.1% -- replaced with /var/log/ceph/ceph-client.admin.160281.log.gz 2023-12-24T21:18:50.549 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170076.log 2023-12-24T21:18:50.549 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152935.log.gz 2023-12-24T21:18:50.549 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122520.log 2023-12-24T21:18:50.549 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125491.log.gz 2023-12-24T21:18:50.549 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.170076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149286.log 2023-12-24T21:18:50.549 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170076.log.gz 2023-12-24T21:18:50.550 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135967.log 2023-12-24T21:18:50.550 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122520.log.gz 2023-12-24T21:18:50.550 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164234.log 2023-12-24T21:18:50.550 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149286.log.gz 2023-12-24T21:18:50.550 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160217.log 2023-12-24T21:18:50.550 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135967.log.gz 2023-12-24T21:18:50.550 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.164234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156190.log 2023-12-24T21:18:50.550 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164234.log.gz 2023-12-24T21:18:50.550 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135039.log 2023-12-24T21:18:50.550 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160217.log.gz 2023-12-24T21:18:50.551 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154035.log 2023-12-24T21:18:50.551 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156190.log.gz 2023-12-24T21:18:50.551 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137712.log 2023-12-24T21:18:50.551 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135039.log.gz 2023-12-24T21:18:50.551 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121219.log 2023-12-24T21:18:50.551 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154035.log.gz 2023-12-24T21:18:50.551 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123789.log 2023-12-24T21:18:50.551 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137712.log.gz 2023-12-24T21:18:50.552 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142861.log 2023-12-24T21:18:50.552 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121219.log.gz 2023-12-24T21:18:50.552 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126069.log 2023-12-24T21:18:50.552 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123789.log.gz 2023-12-24T21:18:50.552 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133047.log 2023-12-24T21:18:50.552 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142861.log.gz 2023-12-24T21:18:50.552 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160711.log 2023-12-24T21:18:50.552 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126069.log.gz 2023-12-24T21:18:50.552 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156566.log 2023-12-24T21:18:50.552 INFO:teuthology.orchestra.run.smithi087.stderr: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.133047.log.gz 2023-12-24T21:18:50.553 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138597.log 2023-12-24T21:18:50.553 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160711.log.gz 2023-12-24T21:18:50.553 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130745.log 2023-12-24T21:18:50.553 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156566.log.gz 2023-12-24T21:18:50.553 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141072.log 2023-12-24T21:18:50.553 INFO:teuthology.orchestra.run.smithi087.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.138597.log.gz 2023-12-24T21:18:50.553 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142307.log 2023-12-24T21:18:50.553 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130745.log.gz 2023-12-24T21:18:50.554 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141072.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141072.log.gz 2023-12-24T21:18:50.554 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152720.log 2023-12-24T21:18:50.554 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142307.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.142307.log.gz--verbose 2023-12-24T21:18:50.554 INFO:teuthology.orchestra.run.smithi087.stderr:-- /var/log/ceph/ceph-client.admin.122910.log 2023-12-24T21:18:50.554 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164360.log 2023-12-24T21:18:50.554 INFO:teuthology.orchestra.run.smithi087.stderr: 11.6% -- replaced with /var/log/ceph/ceph-client.admin.152720.log.gz 2023-12-24T21:18:50.554 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138859.log 2023-12-24T21:18:50.554 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122910.log.gz 2023-12-24T21:18:50.554 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.164360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140895.log 2023-12-24T21:18:50.555 INFO:teuthology.orchestra.run.smithi087.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.164360.log.gz 2023-12-24T21:18:50.555 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131850.log 2023-12-24T21:18:50.555 INFO:teuthology.orchestra.run.smithi087.stderr: 31.5% -- replaced with /var/log/ceph/ceph-client.admin.138859.log.gz 2023-12-24T21:18:50.555 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157314.log 2023-12-24T21:18:50.555 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140895.log.gz 2023-12-24T21:18:50.555 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159452.log 2023-12-24T21:18:50.555 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131850.log.gz 2023-12-24T21:18:50.555 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162856.log 2023-12-24T21:18:50.555 INFO:teuthology.orchestra.run.smithi087.stderr: 57.2% -- replaced with /var/log/ceph/ceph-client.admin.157314.log.gz 2023-12-24T21:18:50.555 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140538.log 2023-12-24T21:18:50.556 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159452.log.gz 2023-12-24T21:18:50.556 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144004.log 2023-12-24T21:18:50.556 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162856.log.gz 2023-12-24T21:18:50.556 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171268.log 2023-12-24T21:18:50.556 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140538.log.gz 2023-12-24T21:18:50.556 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155531.log 2023-12-24T21:18:50.556 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144004.log.gz 2023-12-24T21:18:50.556 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.171268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129073.log 2023-12-24T21:18:50.556 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171268.log.gz 2023-12-24T21:18:50.557 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123939.log 2023-12-24T21:18:50.557 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155531.log.gz 2023-12-24T21:18:50.557 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170574.log 2023-12-24T21:18:50.557 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129073.log.gz 2023-12-24T21:18:50.557 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138378.log 2023-12-24T21:18:50.557 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123939.log.gz 2023-12-24T21:18:50.557 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.170574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144607.log 2023-12-24T21:18:50.557 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170574.log.gz 2023-12-24T21:18:50.558 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123039.log 2023-12-24T21:18:50.558 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138378.log.gz 2023-12-24T21:18:50.558 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158553.log 2023-12-24T21:18:50.558 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144607.log.gz 2023-12-24T21:18:50.558 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162047.log 2023-12-24T21:18:50.558 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123039.log.gz 2023-12-24T21:18:50.558 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142651.log 2023-12-24T21:18:50.558 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158553.log.gz 2023-12-24T21:18:50.558 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160948.log 2023-12-24T21:18:50.558 INFO:teuthology.orchestra.run.smithi087.stderr: 11.1% -- replaced with /var/log/ceph/ceph-client.admin.162047.log.gz 2023-12-24T21:18:50.559 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151203.log 2023-12-24T21:18:50.559 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142651.log.gz 2023-12-24T21:18:50.559 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146263.log 2023-12-24T21:18:50.559 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160948.log.gz 2023-12-24T21:18:50.559 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152958.log 2023-12-24T21:18:50.559 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151203.log.gz 2023-12-24T21:18:50.559 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163399.log 2023-12-24T21:18:50.559 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146263.log.gz 2023-12-24T21:18:50.559 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159904.log 2023-12-24T21:18:50.560 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152958.log.gz 2023-12-24T21:18:50.560 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153283.log 2023-12-24T21:18:50.560 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163399.log.gz 2023-12-24T21:18:50.560 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159824.log 2023-12-24T21:18:50.560 INFO:teuthology.orchestra.run.smithi087.stderr: 22.3% -- replaced with /var/log/ceph/ceph-client.admin.159904.log.gz 2023-12-24T21:18:50.560 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127724.log 2023-12-24T21:18:50.560 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153283.log.gz 2023-12-24T21:18:50.560 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139393.log 2023-12-24T21:18:50.560 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159824.log.gz 2023-12-24T21:18:50.561 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124605.log 2023-12-24T21:18:50.561 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127724.log.gz 2023-12-24T21:18:50.561 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139393.log: gzip 0.0%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.139393.log.gz-- 2023-12-24T21:18:50.561 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136754.log 2023-12-24T21:18:50.561 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155232.log 2023-12-24T21:18:50.561 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124605.log.gz 2023-12-24T21:18:50.561 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164270.log 2023-12-24T21:18:50.561 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136754.log.gz 2023-12-24T21:18:50.562 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168675.log 2023-12-24T21:18:50.562 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155232.log.gz 2023-12-24T21:18:50.562 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.164270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152759.log 2023-12-24T21:18:50.562 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164270.log.gz 2023-12-24T21:18:50.562 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.168675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162231.log 2023-12-24T21:18:50.562 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168675.log.gz 2023-12-24T21:18:50.562 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141252.log 2023-12-24T21:18:50.562 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152759.log.gz 2023-12-24T21:18:50.562 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146650.log 2023-12-24T21:18:50.563 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162231.log.gz 2023-12-24T21:18:50.563 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140213.log 2023-12-24T21:18:50.563 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141252.log.gz 2023-12-24T21:18:50.563 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153042.log 2023-12-24T21:18:50.563 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146650.log.gz 2023-12-24T21:18:50.563 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135154.log 2023-12-24T21:18:50.563 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140213.log.gz 2023-12-24T21:18:50.563 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134223.log 2023-12-24T21:18:50.563 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153042.log.gz 2023-12-24T21:18:50.564 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147279.log 2023-12-24T21:18:50.564 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135154.log.gz 2023-12-24T21:18:50.564 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147201.log 2023-12-24T21:18:50.564 INFO:teuthology.orchestra.run.smithi087.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.134223.log.gz 2023-12-24T21:18:50.564 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147279.log: gzip -5 0.0%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.147279.log.gz 2023-12-24T21:18:50.564 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153203.log 2023-12-24T21:18:50.564 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146586.log 2023-12-24T21:18:50.564 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147201.log.gz 2023-12-24T21:18:50.564 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130193.log 2023-12-24T21:18:50.565 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153203.log.gz 2023-12-24T21:18:50.565 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143896.log 2023-12-24T21:18:50.565 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146586.log.gz 2023-12-24T21:18:50.565 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142672.log 2023-12-24T21:18:50.565 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130193.log.gz 2023-12-24T21:18:50.565 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148317.log 2023-12-24T21:18:50.565 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143896.log.gz 2023-12-24T21:18:50.565 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157509.log 2023-12-24T21:18:50.565 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142672.log.gz 2023-12-24T21:18:50.566 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156527.log 2023-12-24T21:18:50.566 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148317.log.gz 2023-12-24T21:18:50.566 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163654.log 2023-12-24T21:18:50.566 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157509.log.gz 2023-12-24T21:18:50.566 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143853.log 2023-12-24T21:18:50.566 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156527.log.gz 2023-12-24T21:18:50.566 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151163.log 2023-12-24T21:18:50.566 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163654.log.gz 2023-12-24T21:18:50.566 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129573.log 2023-12-24T21:18:50.567 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143853.log.gz 2023-12-24T21:18:50.567 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135902.log 2023-12-24T21:18:50.567 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151163.log.gz 2023-12-24T21:18:50.567 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167279.log 2023-12-24T21:18:50.567 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129573.log.gz 2023-12-24T21:18:50.567 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124906.log 2023-12-24T21:18:50.567 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135902.log.gz 2023-12-24T21:18:50.567 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.167279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128183.log 2023-12-24T21:18:50.567 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167279.log.gz 2023-12-24T21:18:50.568 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159302.log 2023-12-24T21:18:50.568 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124906.log.gz 2023-12-24T21:18:50.568 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140731.log 2023-12-24T21:18:50.568 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.128183.log.gz 2023-12-24T21:18:50.568 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151725.log 2023-12-24T21:18:50.568 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159302.log.gz 2023-12-24T21:18:50.568 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170174.log 2023-12-24T21:18:50.568 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140731.log.gz 2023-12-24T21:18:50.568 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121628.log 2023-12-24T21:18:50.569 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151725.log.gz 2023-12-24T21:18:50.569 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.170174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139159.log 2023-12-24T21:18:50.569 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170174.log.gz 2023-12-24T21:18:50.569 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163525.log 2023-12-24T21:18:50.569 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121628.log.gz 2023-12-24T21:18:50.569 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120064.log 2023-12-24T21:18:50.569 INFO:teuthology.orchestra.run.smithi087.stderr: 23.5% -- replaced with /var/log/ceph/ceph-client.admin.139159.log.gz 2023-12-24T21:18:50.569 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144671.log 2023-12-24T21:18:50.569 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163525.log.gz 2023-12-24T21:18:50.570 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.120064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124712.log 2023-12-24T21:18:50.570 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120064.log.gz 2023-12-24T21:18:50.570 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158936.log 2023-12-24T21:18:50.570 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144671.log.gz 2023-12-24T21:18:50.570 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131901.log 2023-12-24T21:18:50.570 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124712.log.gz 2023-12-24T21:18:50.570 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145797.log 2023-12-24T21:18:50.570 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158936.log.gz 2023-12-24T21:18:50.570 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153799.log 2023-12-24T21:18:50.571 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131901.log.gz 2023-12-24T21:18:50.571 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150862.log 2023-12-24T21:18:50.571 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145797.log.gz 2023-12-24T21:18:50.571 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160518.log 2023-12-24T21:18:50.571 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153799.log.gz 2023-12-24T21:18:50.571 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154121.log 2023-12-24T21:18:50.571 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150862.log.gz 2023-12-24T21:18:50.571 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156936.log 2023-12-24T21:18:50.571 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160518.log.gz 2023-12-24T21:18:50.571 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143208.log 2023-12-24T21:18:50.572 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154121.log.gz 2023-12-24T21:18:50.572 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158688.log 2023-12-24T21:18:50.572 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156936.log.gz 2023-12-24T21:18:50.572 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154900.log 2023-12-24T21:18:50.572 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143208.log.gz 2023-12-24T21:18:50.572 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140559.log 2023-12-24T21:18:50.572 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158688.log.gz 2023-12-24T21:18:50.572 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144177.log 2023-12-24T21:18:50.572 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154900.log.gz 2023-12-24T21:18:50.573 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140559.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140559.log.gz 2023-12-24T21:18:50.573 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164421.log 2023-12-24T21:18:50.573 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152484.log 2023-12-24T21:18:50.573 INFO:teuthology.orchestra.run.smithi087.stderr: 26.7% -- replaced with /var/log/ceph/ceph-client.admin.144177.log.gz 2023-12-24T21:18:50.573 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.164421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125817.log 2023-12-24T21:18:50.573 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164421.log.gz 2023-12-24T21:18:50.573 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130711.log 2023-12-24T21:18:50.573 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152484.log.gz 2023-12-24T21:18:50.573 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134932.log 2023-12-24T21:18:50.574 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125817.log.gz 2023-12-24T21:18:50.574 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150819.log 2023-12-24T21:18:50.574 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130711.log.gz 2023-12-24T21:18:50.574 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152346.log 2023-12-24T21:18:50.574 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134932.log.gz 2023-12-24T21:18:50.574 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141334.log 2023-12-24T21:18:50.574 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150819.log.gz 2023-12-24T21:18:50.574 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123103.log 2023-12-24T21:18:50.574 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152346.log.gz 2023-12-24T21:18:50.575 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125142.log 2023-12-24T21:18:50.575 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141334.log.gz 2023-12-24T21:18:50.575 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146564.log 2023-12-24T21:18:50.575 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123103.log.gz 2023-12-24T21:18:50.575 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137132.log 2023-12-24T21:18:50.575 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125142.log.gz 2023-12-24T21:18:50.575 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151704.log 2023-12-24T21:18:50.575 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146564.log.gz 2023-12-24T21:18:50.575 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136203.log 2023-12-24T21:18:50.576 INFO:teuthology.orchestra.run.smithi087.stderr: 57.1% -- replaced with /var/log/ceph/ceph-client.admin.137132.log.gz 2023-12-24T21:18:50.576 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143724.log 2023-12-24T21:18:50.576 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151704.log.gz 2023-12-24T21:18:50.576 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150625.log 2023-12-24T21:18:50.576 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136203.log.gz 2023-12-24T21:18:50.576 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121735.log 2023-12-24T21:18:50.576 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143724.log.gz 2023-12-24T21:18:50.576 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123318.log 2023-12-24T21:18:50.576 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150625.log.gz 2023-12-24T21:18:50.577 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129613.log 2023-12-24T21:18:50.577 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121735.log.gz 2023-12-24T21:18:50.577 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169128.log 2023-12-24T21:18:50.577 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123318.log.gz 2023-12-24T21:18:50.577 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141107.log 2023-12-24T21:18:50.577 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129613.log.gz 2023-12-24T21:18:50.577 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.169128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151303.log 2023-12-24T21:18:50.577 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169128.log.gz 2023-12-24T21:18:50.577 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147546.log 2023-12-24T21:18:50.578 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141107.log.gz 2023-12-24T21:18:50.578 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130779.log 2023-12-24T21:18:50.578 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151303.log.gz 2023-12-24T21:18:50.578 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119476.log 2023-12-24T21:18:50.578 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147546.log.gz 2023-12-24T21:18:50.578 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133740.log 2023-12-24T21:18:50.578 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130779.log.gz 2023-12-24T21:18:50.578 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.119476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172370.log 2023-12-24T21:18:50.578 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119476.log.gz 2023-12-24T21:18:50.579 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145819.log 2023-12-24T21:18:50.579 INFO:teuthology.orchestra.run.smithi087.stderr: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.133740.log.gz 2023-12-24T21:18:50.579 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.172370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131629.log 2023-12-24T21:18:50.579 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.172370.log.gz 2023-12-24T21:18:50.579 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162705.log 2023-12-24T21:18:50.579 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145819.log.gz 2023-12-24T21:18:50.579 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131629.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131629.log.gz 2023-12-24T21:18:50.579 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153777.log 2023-12-24T21:18:50.579 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162705.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162705.log.gz 2023-12-24T21:18:50.580 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124863.log 2023-12-24T21:18:50.580 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157412.log 2023-12-24T21:18:50.580 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153777.log.gz 2023-12-24T21:18:50.580 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163762.log 2023-12-24T21:18:50.580 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124863.log.gz 2023-12-24T21:18:50.580 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148661.log 2023-12-24T21:18:50.580 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157412.log.gz 2023-12-24T21:18:50.580 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141729.log 2023-12-24T21:18:50.580 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163762.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163762.log.gz 2023-12-24T21:18:50.581 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163482.log 2023-12-24T21:18:50.581 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148661.log.gz 2023-12-24T21:18:50.581 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150711.log 2023-12-24T21:18:50.581 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141729.log.gz 2023-12-24T21:18:50.581 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136410.log 2023-12-24T21:18:50.581 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163482.log.gz 2023-12-24T21:18:50.581 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130013.log 2023-12-24T21:18:50.581 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150711.log.gz 2023-12-24T21:18:50.582 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159280.log 2023-12-24T21:18:50.582 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136410.log.gz 2023-12-24T21:18:50.582 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152286.log 2023-12-24T21:18:50.582 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130013.log.gz 2023-12-24T21:18:50.582 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159280.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159280.log.gz 2023-12-24T21:18:50.582 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157566.log 2023-12-24T21:18:50.582 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160754.log 2023-12-24T21:18:50.582 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152286.log.gz 2023-12-24T21:18:50.582 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146242.log 2023-12-24T21:18:50.582 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157566.log.gz 2023-12-24T21:18:50.583 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132564.log 2023-12-24T21:18:50.583 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160754.log.gz 2023-12-24T21:18:50.583 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152566.log 2023-12-24T21:18:50.583 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146242.log.gz 2023-12-24T21:18:50.583 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137962.log 2023-12-24T21:18:50.583 INFO:teuthology.orchestra.run.smithi087.stderr: 23.2% -- replaced with /var/log/ceph/ceph-client.admin.132564.log.gz 2023-12-24T21:18:50.583 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129693.log 2023-12-24T21:18:50.583 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152566.log.gz 2023-12-24T21:18:50.583 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135077.log 2023-12-24T21:18:50.584 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137962.log.gz 2023-12-24T21:18:50.584 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136675.log 2023-12-24T21:18:50.584 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129693.log.gz 2023-12-24T21:18:50.584 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137359.log 2023-12-24T21:18:50.584 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135077.log.gz 2023-12-24T21:18:50.584 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147506.log 2023-12-24T21:18:50.584 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136675.log.gz 2023-12-24T21:18:50.584 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133131.log 2023-12-24T21:18:50.584 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137359.log.gz 2023-12-24T21:18:50.585 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155018.log 2023-12-24T21:18:50.585 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147506.log.gz 2023-12-24T21:18:50.585 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160797.log 2023-12-24T21:18:50.585 INFO:teuthology.orchestra.run.smithi087.stderr: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.133131.log.gz 2023-12-24T21:18:50.585 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125572.log 2023-12-24T21:18:50.585 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155018.log.gz 2023-12-24T21:18:50.585 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135409.log 2023-12-24T21:18:50.585 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160797.log.gz 2023-12-24T21:18:50.585 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146715.log 2023-12-24T21:18:50.586 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125572.log.gz 2023-12-24T21:18:50.586 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149940.log 2023-12-24T21:18:50.586 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135409.log.gz 2023-12-24T21:18:50.586 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136617.log 2023-12-24T21:18:50.586 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146715.log.gz 2023-12-24T21:18:50.586 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149940.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149940.log.gz 2023-12-24T21:18:50.586 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126670.log 2023-12-24T21:18:50.586 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136617.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136617.log.gz 2023-12-24T21:18:50.586 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125400.log 2023-12-24T21:18:50.586 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130313.log 2023-12-24T21:18:50.587 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.126670.log.gz 2023-12-24T21:18:50.587 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162152.log 2023-12-24T21:18:50.587 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125400.log.gz 2023-12-24T21:18:50.587 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134868.log 2023-12-24T21:18:50.587 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130313.log.gz 2023-12-24T21:18:50.587 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129113.log 2023-12-24T21:18:50.587 INFO:teuthology.orchestra.run.smithi087.stderr: 23.1% -- replaced with /var/log/ceph/ceph-client.admin.162152.log.gz 2023-12-24T21:18:50.587 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168932.log 2023-12-24T21:18:50.587 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134868.log.gz 2023-12-24T21:18:50.588 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124390.log 2023-12-24T21:18:50.588 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129113.log.gz 2023-12-24T21:18:50.588 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.168932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120556.log 2023-12-24T21:18:50.588 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168932.log.gz 2023-12-24T21:18:50.588 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150107.log 2023-12-24T21:18:50.588 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124390.log.gz 2023-12-24T21:18:50.588 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.120556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121692.log 2023-12-24T21:18:50.588 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120556.log.gz 2023-12-24T21:18:50.588 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155191.log 2023-12-24T21:18:50.589 INFO:teuthology.orchestra.run.smithi087.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.150107.log.gz 2023-12-24T21:18:50.589 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144091.log 2023-12-24T21:18:50.589 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121692.log.gz 2023-12-24T21:18:50.589 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161163.log 2023-12-24T21:18:50.589 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155191.log.gz 2023-12-24T21:18:50.589 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130932.log 2023-12-24T21:18:50.589 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144091.log.gz 2023-12-24T21:18:50.589 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129673.log 2023-12-24T21:18:50.590 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161163.log.gz 2023-12-24T21:18:50.590 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152328.log 2023-12-24T21:18:50.590 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130932.log.gz 2023-12-24T21:18:50.590 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121176.log 2023-12-24T21:18:50.590 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129673.log.gz 2023-12-24T21:18:50.590 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135881.log 2023-12-24T21:18:50.590 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152328.log.gz 2023-12-24T21:18:50.590 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142840.log 2023-12-24T21:18:50.590 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121176.log.gz 2023-12-24T21:18:50.591 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122064.log 2023-12-24T21:18:50.591 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135881.log.gz 2023-12-24T21:18:50.591 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153165.log 2023-12-24T21:18:50.591 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142840.log.gz 2023-12-24T21:18:50.591 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139414.log 2023-12-24T21:18:50.591 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.122064.log.gz 2023-12-24T21:18:50.591 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153165.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153165.log.gz 2023-12-24T21:18:50.591 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149114.log 2023-12-24T21:18:50.591 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139414.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139414.log.gz 2023-12-24T21:18:50.592 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135060.log 2023-12-24T21:18:50.592 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156545.log 2023-12-24T21:18:50.592 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149114.log.gz 2023-12-24T21:18:50.592 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159388.log 2023-12-24T21:18:50.592 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.135060.log.gz 2023-12-24T21:18:50.592 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126262.log 2023-12-24T21:18:50.592 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156545.log.gz 2023-12-24T21:18:50.592 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132186.log 2023-12-24T21:18:50.592 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159388.log.gz 2023-12-24T21:18:50.592 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122633.log 2023-12-24T21:18:50.593 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126262.log.gz 2023-12-24T21:18:50.593 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119674.log 2023-12-24T21:18:50.593 INFO:teuthology.orchestra.run.smithi087.stderr: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.132186.log.gz 2023-12-24T21:18:50.593 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151282.log 2023-12-24T21:18:50.593 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122633.log.gz 2023-12-24T21:18:50.593 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.119674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152680.log 2023-12-24T21:18:50.593 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119674.log.gz 2023-12-24T21:18:50.593 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134803.log 2023-12-24T21:18:50.593 INFO:teuthology.orchestra.run.smithi087.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.151282.log.gz 2023-12-24T21:18:50.594 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167230.log 2023-12-24T21:18:50.594 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152680.log.gz 2023-12-24T21:18:50.594 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123383.log 2023-12-24T21:18:50.594 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134803.log.gz 2023-12-24T21:18:50.594 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.167230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127979.log 2023-12-24T21:18:50.594 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167230.log.gz 2023-12-24T21:18:50.594 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138481.log 2023-12-24T21:18:50.594 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123383.log.gz 2023-12-24T21:18:50.594 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164084.log 2023-12-24T21:18:50.595 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127979.log.gz 2023-12-24T21:18:50.595 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138481.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.138481.log.gz 2023-12-24T21:18:50.595 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125662.log 2023-12-24T21:18:50.595 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.164084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148360.log 2023-12-24T21:18:50.595 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164084.log.gz 2023-12-24T21:18:50.595 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156016.log 2023-12-24T21:18:50.595 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125662.log.gz 2023-12-24T21:18:50.595 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146220.log 2023-12-24T21:18:50.595 INFO:teuthology.orchestra.run.smithi087.stderr: 23.4% -- replaced with /var/log/ceph/ceph-client.admin.148360.log.gz 2023-12-24T21:18:50.596 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129653.log 2023-12-24T21:18:50.596 INFO:teuthology.orchestra.run.smithi087.stderr: 25.7% -- replaced with /var/log/ceph/ceph-client.admin.156016.log.gz 2023-12-24T21:18:50.596 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157767.log 2023-12-24T21:18:50.596 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146220.log.gz 2023-12-24T21:18:50.596 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159864.log 2023-12-24T21:18:50.596 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129653.log.gz 2023-12-24T21:18:50.596 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123060.log 2023-12-24T21:18:50.596 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157767.log.gz 2023-12-24T21:18:50.596 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145496.log 2023-12-24T21:18:50.597 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159864.log.gz 2023-12-24T21:18:50.597 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143168.log 2023-12-24T21:18:50.597 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123060.log.gz 2023-12-24T21:18:50.597 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161798.log 2023-12-24T21:18:50.597 INFO:teuthology.orchestra.run.smithi087.stderr: 28.1% -- replaced with /var/log/ceph/ceph-client.admin.145496.log.gz 2023-12-24T21:18:50.597 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145539.log 2023-12-24T21:18:50.597 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143168.log.gz 2023-12-24T21:18:50.597 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141001.log 2023-12-24T21:18:50.597 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161798.log.gz 2023-12-24T21:18:50.597 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147922.log 2023-12-24T21:18:50.598 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145539.log.gz 2023-12-24T21:18:50.598 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121520.log 2023-12-24T21:18:50.598 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141001.log.gz 2023-12-24T21:18:50.598 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126090.log 2023-12-24T21:18:50.598 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.147922.log.gz 2023-12-24T21:18:50.598 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147161.log 2023-12-24T21:18:50.598 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121520.log.gz 2023-12-24T21:18:50.599 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127792.log 2023-12-24T21:18:50.599 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126090.log.gz 2023-12-24T21:18:50.599 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147161.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147161.log.gz 2023-12-24T21:18:50.599 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148231.log 2023-12-24T21:18:50.599 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155471.log 2023-12-24T21:18:50.599 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127792.log.gz 2023-12-24T21:18:50.599 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155113.log 2023-12-24T21:18:50.599 INFO:teuthology.orchestra.run.smithi087.stderr: 23.1% -- replaced with /var/log/ceph/ceph-client.admin.148231.log.gz 2023-12-24T21:18:50.599 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167679.log 2023-12-24T21:18:50.600 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155471.log.gz 2023-12-24T21:18:50.600 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158212.log 2023-12-24T21:18:50.600 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155113.log.gz 2023-12-24T21:18:50.600 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149544.log 2023-12-24T21:18:50.600 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.167679.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167679.log.gz 2023-12-24T21:18:50.600 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128993.log 2023-12-24T21:18:50.600 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158212.log.gz 2023-12-24T21:18:50.600 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151854.log 2023-12-24T21:18:50.600 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149544.log.gz 2023-12-24T21:18:50.600 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139179.log 2023-12-24T21:18:50.601 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128993.log.gz 2023-12-24T21:18:50.601 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172117.log 2023-12-24T21:18:50.601 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151854.log.gz 2023-12-24T21:18:50.601 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138461.log 2023-12-24T21:18:50.601 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139179.log.gz 2023-12-24T21:18:50.601 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.172117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157529.log 2023-12-24T21:18:50.601 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.172117.log.gz 2023-12-24T21:18:50.601 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131000.log 2023-12-24T21:18:50.601 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138461.log.gz 2023-12-24T21:18:50.602 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140667.log 2023-12-24T21:18:50.602 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157529.log.gz 2023-12-24T21:18:50.602 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168785.log 2023-12-24T21:18:50.602 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131000.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131000.log.gz 2023-12-24T21:18:50.602 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140450.log 2023-12-24T21:18:50.602 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140667.log.gz 2023-12-24T21:18:50.602 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.168785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145132.log 2023-12-24T21:18:50.602 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168785.log.gz 2023-12-24T21:18:50.602 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139672.log 2023-12-24T21:18:50.603 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140450.log.gz 2023-12-24T21:18:50.603 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150408.log 2023-12-24T21:18:50.603 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145132.log.gz 2023-12-24T21:18:50.603 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150961.log 2023-12-24T21:18:50.603 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139672.log.gz 2023-12-24T21:18:50.603 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143151.log 2023-12-24T21:18:50.603 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150408.log.gz 2023-12-24T21:18:50.603 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156759.log 2023-12-24T21:18:50.603 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150961.log.gz 2023-12-24T21:18:50.603 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124841.log 2023-12-24T21:18:50.604 INFO:teuthology.orchestra.run.smithi087.stderr: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.143151.log.gz 2023-12-24T21:18:50.604 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127452.log 2023-12-24T21:18:50.604 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156759.log.gz 2023-12-24T21:18:50.604 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.124841.log: -- /var/log/ceph/ceph-client.admin.127877.log 2023-12-24T21:18:50.604 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124841.log.gz 2023-12-24T21:18:50.604 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166145.log 2023-12-24T21:18:50.604 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127452.log.gz 2023-12-24T21:18:50.604 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155585.log 2023-12-24T21:18:50.604 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127877.log.gz 2023-12-24T21:18:50.604 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.166145.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166145.log.gz 2023-12-24T21:18:50.605 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158915.log 2023-12-24T21:18:50.605 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155585.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155585.log.gz 2023-12-24T21:18:50.605 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144456.log 2023-12-24T21:18:50.605 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122102.log 2023-12-24T21:18:50.605 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158915.log.gz 2023-12-24T21:18:50.605 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139285.log 2023-12-24T21:18:50.605 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144456.log.gz 2023-12-24T21:18:50.605 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127095.log 2023-12-24T21:18:50.605 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.122102.log.gz 2023-12-24T21:18:50.606 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131272.log 2023-12-24T21:18:50.606 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139285.log.gz 2023-12-24T21:18:50.606 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143005.log 2023-12-24T21:18:50.606 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127095.log.gz 2023-12-24T21:18:50.606 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126370.log 2023-12-24T21:18:50.606 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131272.log.gz 2023-12-24T21:18:50.606 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152837.log 2023-12-24T21:18:50.606 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143005.log.gz 2023-12-24T21:18:50.606 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141836.log 2023-12-24T21:18:50.606 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126370.log.gz 2023-12-24T21:18:50.607 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147118.log 2023-12-24T21:18:50.607 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152837.log.gz 2023-12-24T21:18:50.607 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131816.log 2023-12-24T21:18:50.607 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141836.log.gz 2023-12-24T21:18:50.607 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155956.log 2023-12-24T21:18:50.607 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147118.log.gz 2023-12-24T21:18:50.607 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146414.log 2023-12-24T21:18:50.607 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131816.log.gz 2023-12-24T21:18:50.607 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130456.log 2023-12-24T21:18:50.608 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155956.log.gz 2023-12-24T21:18:50.608 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161141.log 2023-12-24T21:18:50.608 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146414.log.gz 2023-12-24T21:18:50.608 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139737.log 2023-12-24T21:18:50.608 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130456.log.gz 2023-12-24T21:18:50.608 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136160.log 2023-12-24T21:18:50.608 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161141.log.gz 2023-12-24T21:18:50.608 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164505.log 2023-12-24T21:18:50.608 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139737.log.gz 2023-12-24T21:18:50.608 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131935.log 2023-12-24T21:18:50.609 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136160.log.gz 2023-12-24T21:18:50.609 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.164505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170525.log 2023-12-24T21:18:50.609 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164505.log.gz 2023-12-24T21:18:50.609 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160733.log 2023-12-24T21:18:50.609 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131935.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131935.log.gz 2023-12-24T21:18:50.609 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.170525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122760.log 2023-12-24T21:18:50.609 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170525.log.gz 2023-12-24T21:18:50.609 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160733.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160733.log.gz 2023-12-24T21:18:50.609 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147321.log 2023-12-24T21:18:50.610 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142529.log 2023-12-24T21:18:50.610 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122760.log.gz 2023-12-24T21:18:50.610 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147321.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147321.log.gz 2023-12-24T21:18:50.610 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128833.log 2023-12-24T21:18:50.610 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171023.log 2023-12-24T21:18:50.610 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142529.log.gz 2023-12-24T21:18:50.610 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151919.log 2023-12-24T21:18:50.610 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128833.log.gz 2023-12-24T21:18:50.610 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.171023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138680.log 2023-12-24T21:18:50.610 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171023.log.gz 2023-12-24T21:18:50.611 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128913.log 2023-12-24T21:18:50.611 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151919.log.gz 2023-12-24T21:18:50.611 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145883.log 2023-12-24T21:18:50.611 INFO:teuthology.orchestra.run.smithi087.stderr: 21.5% -- replaced with /var/log/ceph/ceph-client.admin.138680.log.gz 2023-12-24T21:18:50.611 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126738.log 2023-12-24T21:18:50.611 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128913.log.gz 2023-12-24T21:18:50.611 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142881.log 2023-12-24T21:18:50.611 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145883.log.gz 2023-12-24T21:18:50.611 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137196.log 2023-12-24T21:18:50.611 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.126738.log.gz 2023-12-24T21:18:50.612 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163440.log 2023-12-24T21:18:50.612 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142881.log.gz 2023-12-24T21:18:50.612 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137196.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137196.log.gz 2023-12-24T21:18:50.612 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157726.log 2023-12-24T21:18:50.612 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136031.log 2023-12-24T21:18:50.612 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163440.log.gz 2023-12-24T21:18:50.612 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149876.log 2023-12-24T21:18:50.612 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157726.log.gz 2023-12-24T21:18:50.612 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144628.log 2023-12-24T21:18:50.612 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136031.log.gz 2023-12-24T21:18:50.613 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146138.log 2023-12-24T21:18:50.613 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149876.log.gz 2023-12-24T21:18:50.613 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156307.log 2023-12-24T21:18:50.613 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144628.log.gz 2023-12-24T21:18:50.613 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154143.log 2023-12-24T21:18:50.613 INFO:teuthology.orchestra.run.smithi087.stderr: 56.2% -- replaced with /var/log/ceph/ceph-client.admin.146138.log.gz 2023-12-24T21:18:50.613 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142775.log 2023-12-24T21:18:50.613 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156307.log.gz 2023-12-24T21:18:50.613 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137464.log 2023-12-24T21:18:50.614 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154143.log.gz 2023-12-24T21:18:50.614 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138263.log 2023-12-24T21:18:50.614 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142775.log.gz 2023-12-24T21:18:50.614 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163122.log 2023-12-24T21:18:50.614 INFO:teuthology.orchestra.run.smithi087.stderr: 57.1% -- replaced with /var/log/ceph/ceph-client.admin.137464.log.gz 2023-12-24T21:18:50.614 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139522.log 2023-12-24T21:18:50.614 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138263.log.gz 2023-12-24T21:18:50.614 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147880.log 2023-12-24T21:18:50.614 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163122.log.gz 2023-12-24T21:18:50.614 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125271.log 2023-12-24T21:18:50.615 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139522.log.gz 2023-12-24T21:18:50.615 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151242.log 2023-12-24T21:18:50.615 INFO:teuthology.orchestra.run.smithi087.stderr: 23.1% -- replaced with /var/log/ceph/ceph-client.admin.147880.log.gz 2023-12-24T21:18:50.615 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170778.log 2023-12-24T21:18:50.615 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125271.log.gz 2023-12-24T21:18:50.615 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131612.log 2023-12-24T21:18:50.615 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151242.log.gz 2023-12-24T21:18:50.615 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.170778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124497.log 2023-12-24T21:18:50.615 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170778.log.gz 2023-12-24T21:18:50.615 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130762.log 2023-12-24T21:18:50.616 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131612.log.gz 2023-12-24T21:18:50.616 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124497.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124497.log.gz 2023-12-24T21:18:50.616 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171578.log 2023-12-24T21:18:50.616 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130762.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130762.log.gz 2023-12-24T21:18:50.616 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121757.log 2023-12-24T21:18:50.616 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.171578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132942.log 2023-12-24T21:18:50.616 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171578.log.gz 2023-12-24T21:18:50.616 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124111.log 2023-12-24T21:18:50.616 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121757.log.gz 2023-12-24T21:18:50.617 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168626.log 2023-12-24T21:18:50.617 INFO:teuthology.orchestra.run.smithi087.stderr: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.132942.log.gz 2023-12-24T21:18:50.617 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168177.log 2023-12-24T21:18:50.617 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124111.log.gz 2023-12-24T21:18:50.617 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.168626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150538.log 2023-12-24T21:18:50.617 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168626.log.gz 2023-12-24T21:18:50.617 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.168177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158669.log 2023-12-24T21:18:50.617 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168177.log.gz 2023-12-24T21:18:50.617 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128693.log 2023-12-24T21:18:50.618 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150538.log.gz 2023-12-24T21:18:50.618 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141858.log 2023-12-24T21:18:50.618 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158669.log.gz 2023-12-24T21:18:50.618 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144025.log 2023-12-24T21:18:50.618 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128693.log.gz 2023-12-24T21:18:50.618 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158786.log 2023-12-24T21:18:50.618 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141858.log.gz 2023-12-24T21:18:50.618 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166576.log 2023-12-24T21:18:50.618 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144025.log.gz 2023-12-24T21:18:50.618 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128098.log 2023-12-24T21:18:50.619 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158786.log.gz 2023-12-24T21:18:50.619 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.166576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156819.log 2023-12-24T21:18:50.619 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166576.log.gz 2023-12-24T21:18:50.619 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128217.log 2023-12-24T21:18:50.619 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.128098.log.gz 2023-12-24T21:18:50.619 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156819.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156819.log.gz 2023-12-24T21:18:50.619 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146435.log 2023-12-24T21:18:50.619 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147796.log 2023-12-24T21:18:50.619 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128217.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.128217.log.gz 2023-12-24T21:18:50.620 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141055.log 2023-12-24T21:18:50.620 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146435.log.gz 2023-12-24T21:18:50.620 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148446.log 2023-12-24T21:18:50.620 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147796.log.gz 2023-12-24T21:18:50.620 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142921.log 2023-12-24T21:18:50.620 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141055.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141055.log.gz 2023-12-24T21:18:50.620 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131765.log 2023-12-24T21:18:50.620 INFO:teuthology.orchestra.run.smithi087.stderr: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.148446.log.gz 2023-12-24T21:18:50.620 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147632.log 2023-12-24T21:18:50.620 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142921.log.gz 2023-12-24T21:18:50.621 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163461.log 2023-12-24T21:18:50.621 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131765.log.gz 2023-12-24T21:18:50.621 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130524.log 2023-12-24T21:18:50.621 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147632.log.gz 2023-12-24T21:18:50.621 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163461.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163461.log.gz 2023-12-24T21:18:50.621 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126584.log 2023-12-24T21:18:50.621 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149415.log 2023-12-24T21:18:50.621 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130524.log.gz 2023-12-24T21:18:50.621 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126584.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.126584.log.gz--verbose 2023-12-24T21:18:50.622 INFO:teuthology.orchestra.run.smithi087.stderr:-- /var/log/ceph/ceph-client.admin.143311.log 2023-12-24T21:18:50.622 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127044.log 2023-12-24T21:18:50.622 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149415.log.gz 2023-12-24T21:18:50.622 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143311.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143311.log.gz 2023-12-24T21:18:50.622 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136139.log 2023-12-24T21:18:50.622 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169626.log 2023-12-24T21:18:50.622 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127044.log.gz 2023-12-24T21:18:50.622 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136139.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.136139.log.gz-5 2023-12-24T21:18:50.622 INFO:teuthology.orchestra.run.smithi087.stderr:--verbose -- /var/log/ceph/ceph-client.admin.149458.log 2023-12-24T21:18:50.622 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.169626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129033.log 2023-12-24T21:18:50.623 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169626.log.gz 2023-12-24T21:18:50.623 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140106.log 2023-12-24T21:18:50.623 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149458.log.gz 2023-12-24T21:18:50.623 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133992.log 2023-12-24T21:18:50.623 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129033.log.gz 2023-12-24T21:18:50.623 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147238.log 2023-12-24T21:18:50.623 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140106.log.gz 2023-12-24T21:18:50.623 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162985.log 2023-12-24T21:18:50.623 INFO:teuthology.orchestra.run.smithi087.stderr: 23.4% -- replaced with /var/log/ceph/ceph-client.admin.133992.log.gz 2023-12-24T21:18:50.623 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147238.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.138419.log 2023-12-24T21:18:50.624 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.147238.log.gz 2023-12-24T21:18:50.624 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155997.log 2023-12-24T21:18:50.624 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162985.log.gz 2023-12-24T21:18:50.624 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170419.log 2023-12-24T21:18:50.624 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138419.log.gz 2023-12-24T21:18:50.624 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143130.log 2023-12-24T21:18:50.624 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155997.log.gz 2023-12-24T21:18:50.624 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.170419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149835.log 2023-12-24T21:18:50.624 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170419.log.gz 2023-12-24T21:18:50.624 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143130.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143130.log.gz 2023-12-24T21:18:50.625 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136451.log 2023-12-24T21:18:50.625 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129713.log 2023-12-24T21:18:50.625 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149835.log.gz 2023-12-24T21:18:50.625 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160238.log 2023-12-24T21:18:50.625 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136451.log.gz 2023-12-24T21:18:50.625 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144736.log 2023-12-24T21:18:50.625 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129713.log.gz 2023-12-24T21:18:50.625 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131187.log 2023-12-24T21:18:50.625 INFO:teuthology.orchestra.run.smithi087.stderr: 23.9% -- replaced with /var/log/ceph/ceph-client.admin.160238.log.gz 2023-12-24T21:18:50.626 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144736.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144736.log.gz 2023-12-24T21:18:50.626 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122698.log 2023-12-24T21:18:50.626 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150647.log 2023-12-24T21:18:50.626 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131187.log.gz 2023-12-24T21:18:50.626 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148575.log 2023-12-24T21:18:50.626 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122698.log.gz 2023-12-24T21:18:50.626 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146736.log 2023-12-24T21:18:50.626 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150647.log.gz 2023-12-24T21:18:50.626 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139121.log 2023-12-24T21:18:50.627 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148575.log.gz 2023-12-24T21:18:50.627 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151356.log 2023-12-24T21:18:50.627 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146736.log.gz 2023-12-24T21:18:50.627 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119673.log 2023-12-24T21:18:50.627 INFO:teuthology.orchestra.run.smithi087.stderr: 23.1% -- replaced with /var/log/ceph/ceph-client.admin.139121.log.gz 2023-12-24T21:18:50.627 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127146.log 2023-12-24T21:18:50.627 INFO:teuthology.orchestra.run.smithi087.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.151356.log.gz 2023-12-24T21:18:50.627 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.119673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133299.log 2023-12-24T21:18:50.627 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119673.log.gz 2023-12-24T21:18:50.628 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158872.log 2023-12-24T21:18:50.628 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127146.log.gz 2023-12-24T21:18:50.628 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152874.log 2023-12-24T21:18:50.628 INFO:teuthology.orchestra.run.smithi087.stderr: 22.3% -- replaced with /var/log/ceph/ceph-client.admin.133299.log.gz 2023-12-24T21:18:50.628 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159474.log 2023-12-24T21:18:50.628 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158872.log.gz 2023-12-24T21:18:50.628 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121499.log 2023-12-24T21:18:50.628 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152874.log.gz 2023-12-24T21:18:50.628 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138060.log 2023-12-24T21:18:50.629 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159474.log.gz 2023-12-24T21:18:50.629 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158252.log 2023-12-24T21:18:50.629 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121499.log.gz 2023-12-24T21:18:50.629 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122802.log 2023-12-24T21:18:50.629 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138060.log.gz 2023-12-24T21:18:50.629 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120211.log 2023-12-24T21:18:50.629 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158252.log.gz 2023-12-24T21:18:50.629 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160991.log 2023-12-24T21:18:50.629 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122802.log.gz 2023-12-24T21:18:50.630 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.120211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144155.log 2023-12-24T21:18:50.630 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120211.log.gz 2023-12-24T21:18:50.630 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157489.log 2023-12-24T21:18:50.630 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160991.log.gz 2023-12-24T21:18:50.630 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166390.log 2023-12-24T21:18:50.630 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144155.log.gz 2023-12-24T21:18:50.630 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128533.log 2023-12-24T21:18:50.630 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157489.log.gz 2023-12-24T21:18:50.630 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.166390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164341.log 2023-12-24T21:18:50.630 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166390.log.gz 2023-12-24T21:18:50.631 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130660.log 2023-12-24T21:18:50.631 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128533.log.gz 2023-12-24T21:18:50.631 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.164341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114655.log 2023-12-24T21:18:50.631 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164341.log.gz 2023-12-24T21:18:50.631 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161818.log 2023-12-24T21:18:50.631 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130660.log.gz 2023-12-24T21:18:50.631 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.114655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147672.log 2023-12-24T21:18:50.631 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114655.log.gz 2023-12-24T21:18:50.631 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131459.log 2023-12-24T21:18:50.631 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161818.log.gz 2023-12-24T21:18:50.632 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130847.log 2023-12-24T21:18:50.632 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147672.log.gz 2023-12-24T21:18:50.632 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131459.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-12-24T21:18:50.632 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131459.log.gz 2023-12-24T21:18:50.632 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161838.log 2023-12-24T21:18:50.632 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130847.log.gz 2023-12-24T21:18:50.632 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142287.log 2023-12-24T21:18:50.632 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138040.log 2023-12-24T21:18:50.632 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161838.log.gz 2023-12-24T21:18:50.633 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125056.log 2023-12-24T21:18:50.633 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142287.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142287.log.gz 2023-12-24T21:18:50.633 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138040.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.159108.log 2023-12-24T21:18:50.633 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.138040.log.gz 2023-12-24T21:18:50.633 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142327.log 2023-12-24T21:18:50.633 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125056.log.gz 2023-12-24T21:18:50.633 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159783.log 2023-12-24T21:18:50.633 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159108.log.gz 2023-12-24T21:18:50.633 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142327.log: 0.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.142327.log.gz--verbose 2023-12-24T21:18:50.634 INFO:teuthology.orchestra.run.smithi087.stderr:-- /var/log/ceph/ceph-client.admin.169471.log 2023-12-24T21:18:50.634 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137547.log 2023-12-24T21:18:50.634 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159783.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159783.log.gz 2023-12-24T21:18:50.634 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.169471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148856.log 2023-12-24T21:18:50.634 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169471.log.gz 2023-12-24T21:18:50.634 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137547.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137547.log.gz 2023-12-24T21:18:50.634 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168520.log 2023-12-24T21:18:50.634 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152265.log 2023-12-24T21:18:50.634 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148856.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148856.log.gz 2023-12-24T21:18:50.635 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.168520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122678.log 2023-12-24T21:18:50.635 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168520.log.gz 2023-12-24T21:18:50.635 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153455.log 2023-12-24T21:18:50.635 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152265.log.gz 2023-12-24T21:18:50.635 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151833.log 2023-12-24T21:18:50.635 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122678.log.gz 2023-12-24T21:18:50.635 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158589.log 2023-12-24T21:18:50.635 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153455.log.gz 2023-12-24T21:18:50.635 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151833.log: 0.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.151833.log.gz--verbose 2023-12-24T21:18:50.635 INFO:teuthology.orchestra.run.smithi087.stderr:-- /var/log/ceph/ceph-client.admin.129353.log 2023-12-24T21:18:50.636 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121714.log 2023-12-24T21:18:50.636 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158589.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158589.log.gz 2023-12-24T21:18:50.636 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153605.log 2023-12-24T21:18:50.636 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129353.log.gz 2023-12-24T21:18:50.636 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121714.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121714.log.gzgzip 2023-12-24T21:18:50.636 INFO:teuthology.orchestra.run.smithi087.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.139200.log 2023-12-24T21:18:50.636 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162211.log 2023-12-24T21:18:50.636 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153605.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153605.log.gz 2023-12-24T21:18:50.636 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122330.log 2023-12-24T21:18:50.637 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139200.log.gz 2023-12-24T21:18:50.637 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120458.log 2023-12-24T21:18:50.637 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162211.log.gz 2023-12-24T21:18:50.637 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124648.log 2023-12-24T21:18:50.637 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122330.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.122330.log.gz 2023-12-24T21:18:50.637 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.120458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150129.log 2023-12-24T21:18:50.637 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120458.log.gz 2023-12-24T21:18:50.637 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166341.log 2023-12-24T21:18:50.637 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124648.log.gz 2023-12-24T21:18:50.638 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157626.log 2023-12-24T21:18:50.638 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150129.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150129.log.gz 2023-12-24T21:18:50.638 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.166341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148747.log 2023-12-24T21:18:50.638 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166341.log.gz 2023-12-24T21:18:50.638 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157626.log: 0.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.157626.log.gz--verbose 2023-12-24T21:18:50.638 INFO:teuthology.orchestra.run.smithi087.stderr:-- /var/log/ceph/ceph-client.admin.160026.log 2023-12-24T21:18:50.638 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152641.log 2023-12-24T21:18:50.638 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148747.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148747.log.gz 2023-12-24T21:18:50.638 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121869.log 2023-12-24T21:18:50.638 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160026.log.gz 2023-12-24T21:18:50.639 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152641.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152641.log.gz 2023-12-24T21:18:50.639 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137649.log 2023-12-24T21:18:50.639 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141294.log 2023-12-24T21:18:50.639 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121869.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121869.log.gz 2023-12-24T21:18:50.639 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162089.log 2023-12-24T21:18:50.639 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137649.log.gz 2023-12-24T21:18:50.639 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148532.log 2023-12-24T21:18:50.639 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141294.log.gz 2023-12-24T21:18:50.639 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122043.log 2023-12-24T21:18:50.640 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162089.log: 11.1% -- replaced with /var/log/ceph/ceph-client.admin.162089.log.gz 2023-12-24T21:18:50.640 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142467.log 2023-12-24T21:18:50.640 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148532.log.gz 2023-12-24T21:18:50.640 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122043.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122043.log.gz 2023-12-24T21:18:50.640 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159044.log 2023-12-24T21:18:50.640 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172264.log 2023-12-24T21:18:50.640 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142467.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142467.log.gz 2023-12-24T21:18:50.640 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129593.log 2023-12-24T21:18:50.640 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159044.log.gz 2023-12-24T21:18:50.640 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.172264.log: gzip 0.0%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.172264.log.gz-- 2023-12-24T21:18:50.641 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.167728.log 2023-12-24T21:18:50.641 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127775.log 2023-12-24T21:18:50.641 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129593.log.gz 2023-12-24T21:18:50.641 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.167728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131408.log 2023-12-24T21:18:50.641 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167728.log.gz 2023-12-24T21:18:50.641 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163568.log 2023-12-24T21:18:50.641 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127775.log.gz 2023-12-24T21:18:50.641 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126305.log 2023-12-24T21:18:50.641 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131408.log.gz 2023-12-24T21:18:50.642 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153756.log 2023-12-24T21:18:50.642 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163568.log.gz 2023-12-24T21:18:50.642 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140364.log 2023-12-24T21:18:50.642 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126305.log.gz 2023-12-24T21:18:50.642 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159409.log 2023-12-24T21:18:50.642 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153756.log.gz 2023-12-24T21:18:50.642 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158232.log 2023-12-24T21:18:50.642 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140364.log.gz 2023-12-24T21:18:50.642 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149092.log 2023-12-24T21:18:50.642 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159409.log.gz 2023-12-24T21:18:50.643 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128234.log 2023-12-24T21:18:50.643 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158232.log.gz 2023-12-24T21:18:50.643 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129433.log 2023-12-24T21:18:50.643 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149092.log.gz 2023-12-24T21:18:50.643 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136010.log 2023-12-24T21:18:50.643 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.128234.log.gz 2023-12-24T21:18:50.643 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142631.log 2023-12-24T21:18:50.643 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129433.log.gz 2023-12-24T21:18:50.643 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137815.log 2023-12-24T21:18:50.644 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136010.log.gz 2023-12-24T21:18:50.644 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127928.log 2023-12-24T21:18:50.644 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142631.log.gz 2023-12-24T21:18:50.644 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147959.log 2023-12-24T21:18:50.644 INFO:teuthology.orchestra.run.smithi087.stderr: 23.5% -- replaced with /var/log/ceph/ceph-client.admin.137815.log.gz 2023-12-24T21:18:50.644 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157881.log 2023-12-24T21:18:50.644 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127928.log.gz 2023-12-24T21:18:50.644 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163355.log 2023-12-24T21:18:50.644 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147959.log.gz 2023-12-24T21:18:50.644 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157881.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157881.log.gz 2023-12-24T21:18:50.645 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140930.log 2023-12-24T21:18:50.645 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151489.log 2023-12-24T21:18:50.645 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163355.log: 55.9% -- replaced with /var/log/ceph/ceph-client.admin.163355.log.gz 2023-12-24T21:18:50.645 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140930.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140930.log.gz 2023-12-24T21:18:50.645 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132858.log 2023-12-24T21:18:50.645 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151489.log.gz 2023-12-24T21:18:50.645 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162555.log 2023-12-24T21:18:50.645 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164445.log 2023-12-24T21:18:50.645 INFO:teuthology.orchestra.run.smithi087.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.132858.log.gz 2023-12-24T21:18:50.646 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122019.log 2023-12-24T21:18:50.646 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162555.log.gz 2023-12-24T21:18:50.646 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.164445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129493.log 2023-12-24T21:18:50.646 INFO:teuthology.orchestra.run.smithi087.stderr: 54.5% -- replaced with /var/log/ceph/ceph-client.admin.164445.log.gz 2023-12-24T21:18:50.646 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135250.log 2023-12-24T21:18:50.646 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122019.log.gz 2023-12-24T21:18:50.646 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126704.log 2023-12-24T21:18:50.646 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129493.log.gz 2023-12-24T21:18:50.646 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162963.log 2023-12-24T21:18:50.647 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135250.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135250.log.gz 2023-12-24T21:18:50.647 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126704.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.126704.log.gz 2023-12-24T21:18:50.647 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162963.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162963.log.gz 2023-12-24T21:18:50.647 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126687.log 2023-12-24T21:18:50.647 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161657.log 2023-12-24T21:18:50.647 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153243.log 2023-12-24T21:18:50.647 INFO:teuthology.orchestra.run.smithi087.stderr: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.126687.log.gz 2023-12-24T21:18:50.647 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157017.log /var/log/ceph/ceph-client.admin.161657.log: 2023-12-24T21:18:50.647 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161657.log.gz 2023-12-24T21:18:50.647 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129553.log 2023-12-24T21:18:50.648 INFO:teuthology.orchestra.run.smithi087.stderr: 11.6% -- replaced with /var/log/ceph/ceph-client.admin.153243.log.gz 2023-12-24T21:18:50.648 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135173.log 2023-12-24T21:18:50.648 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157017.log.gz 2023-12-24T21:18:50.648 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149897.log 2023-12-24T21:18:50.648 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129553.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129553.log.gz 2023-12-24T21:18:50.648 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135173.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.135173.log.gz 2023-12-24T21:18:50.648 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149897.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149897.log.gz 2023-12-24T21:18:50.648 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119378.log 2023-12-24T21:18:50.648 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158312.log 2023-12-24T21:18:50.649 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.119378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143660.log 2023-12-24T21:18:50.649 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119378.log.gz 2023-12-24T21:18:50.649 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171121.log 2023-12-24T21:18:50.649 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158312.log.gz 2023-12-24T21:18:50.649 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130643.log 2023-12-24T21:18:50.649 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143660.log.gz 2023-12-24T21:18:50.649 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.171121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132984.log 2023-12-24T21:18:50.649 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171121.log.gz 2023-12-24T21:18:50.649 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137095.log 2023-12-24T21:18:50.650 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130643.log.gz 2023-12-24T21:18:50.650 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147218.log 2023-12-24T21:18:50.650 INFO:teuthology.orchestra.run.smithi087.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.132984.log.gz 2023-12-24T21:18:50.650 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141211.log 2023-12-24T21:18:50.650 INFO:teuthology.orchestra.run.smithi087.stderr: 58.5% -- replaced with /var/log/ceph/ceph-client.admin.137095.log.gz 2023-12-24T21:18:50.650 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129473.log 2023-12-24T21:18:50.650 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147218.log.gz 2023-12-24T21:18:50.650 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160969.log 2023-12-24T21:18:50.650 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141211.log.gz 2023-12-24T21:18:50.650 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127163.log 2023-12-24T21:18:50.651 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129473.log.gz 2023-12-24T21:18:50.651 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150322.log 2023-12-24T21:18:50.651 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160969.log.gz 2023-12-24T21:18:50.651 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166780.log 2023-12-24T21:18:50.651 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127163.log.gz 2023-12-24T21:18:50.651 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123588.log 2023-12-24T21:18:50.651 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150322.log: 26.0% -- replaced with /var/log/ceph/ceph-client.admin.150322.log.gz 2023-12-24T21:18:50.651 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.166780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128336.log 2023-12-24T21:18:50.651 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166780.log.gz 2023-12-24T21:18:50.652 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123588.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123588.log.gz 2023-12-24T21:18:50.652 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152662.log 2023-12-24T21:18:50.652 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133656.log 2023-12-24T21:18:50.652 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.128336.log.gz 2023-12-24T21:18:50.652 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153928.log 2023-12-24T21:18:50.652 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152662.log.gz 2023-12-24T21:18:50.652 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148253.log 2023-12-24T21:18:50.652 INFO:teuthology.orchestra.run.smithi087.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.133656.log.gz 2023-12-24T21:18:50.652 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167777.log 2023-12-24T21:18:50.652 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153928.log.gz 2023-12-24T21:18:50.653 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158078.log 2023-12-24T21:18:50.653 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148253.log.gz 2023-12-24T21:18:50.653 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.167777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144923.log 2023-12-24T21:18:50.653 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167777.log.gz 2023-12-24T21:18:50.653 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158078.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158078.log.gz 2023-12-24T21:18:50.653 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128115.log 2023-12-24T21:18:50.653 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136954.log 2023-12-24T21:18:50.653 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144923.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144923.log.gz 2023-12-24T21:18:50.653 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159636.log 2023-12-24T21:18:50.654 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.128115.log.gz 2023-12-24T21:18:50.654 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143382.log 2023-12-24T21:18:50.654 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136954.log.gz 2023-12-24T21:18:50.654 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128370.log 2023-12-24T21:18:50.654 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159636.log.gz 2023-12-24T21:18:50.654 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143382.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143382.log.gz 2023-12-24T21:18:50.654 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131714.log 2023-12-24T21:18:50.654 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143617.log 2023-12-24T21:18:50.654 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128370.log.gz 2023-12-24T21:18:50.654 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121824.log 2023-12-24T21:18:50.655 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131714.log.gz 2023-12-24T21:18:50.655 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143617.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.143617.log.gz--verbose 2023-12-24T21:18:50.655 INFO:teuthology.orchestra.run.smithi087.stderr:-- /var/log/ceph/ceph-client.admin.138498.log 2023-12-24T21:18:50.655 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.121824.log: -- /var/log/ceph/ceph-client.admin.154616.log 2023-12-24T21:18:50.655 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121824.log.gz 2023-12-24T21:18:50.655 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138821.log 2023-12-24T21:18:50.655 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138498.log.gz 2023-12-24T21:18:50.655 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154616.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154616.log.gz 2023-12-24T21:18:50.655 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138821.log: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.138821.log.gz 2023-12-24T21:18:50.656 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151790.log 2023-12-24T21:18:50.656 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145432.log 2023-12-24T21:18:50.656 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151790.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151790.log.gz 2023-12-24T21:18:50.656 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129193.log 2023-12-24T21:18:50.656 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156431.log 2023-12-24T21:18:50.656 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145432.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145432.log.gz 2023-12-24T21:18:50.656 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129193.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129193.log.gz 2023-12-24T21:18:50.656 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156431.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156431.log.gz 2023-12-24T21:18:50.656 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171423.log 2023-12-24T21:18:50.656 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137941.log 2023-12-24T21:18:50.657 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.171423.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171423.log.gz 2023-12-24T21:18:50.657 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167426.log 2023-12-24T21:18:50.657 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139242.log 2023-12-24T21:18:50.657 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137941.log: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.137941.log.gz 2023-12-24T21:18:50.657 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.167426.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167426.log.gz 2023-12-24T21:18:50.657 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139242.log: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.139242.log.gz 2023-12-24T21:18:50.657 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163314.log 2023-12-24T21:18:50.657 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158155.log 2023-12-24T21:18:50.657 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164343.log 2023-12-24T21:18:50.658 INFO:teuthology.orchestra.run.smithi087.stderr: 53.5% -- replaced with /var/log/ceph/ceph-client.admin.163314.log.gz 2023-12-24T21:18:50.658 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124411.log 2023-12-24T21:18:50.658 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158155.log.gz 2023-12-24T21:18:50.658 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.164343.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164343.log.gz 2023-12-24T21:18:50.658 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161904.log 2023-12-24T21:18:50.658 INFO:teuthology.orchestra.run.smithi087.stderr:gzip /var/log/ceph/ceph-client.admin.124411.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.135348.log 2023-12-24T21:18:50.658 INFO:teuthology.orchestra.run.smithi087.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.124411.log.gz 2023-12-24T21:18:50.658 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155684.log 2023-12-24T21:18:50.658 INFO:teuthology.orchestra.run.smithi087.stderr: 23.4% -- replaced with /var/log/ceph/ceph-client.admin.161904.log.gz 2023-12-24T21:18:50.658 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169226.log 2023-12-24T21:18:50.659 INFO:teuthology.orchestra.run.smithi087.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.135348.log.gz 2023-12-24T21:18:50.659 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137627.log 2023-12-24T21:18:50.659 INFO:teuthology.orchestra.run.smithi087.stderr: 53.7% -- replaced with /var/log/ceph/ceph-client.admin.155684.log.gz 2023-12-24T21:18:50.659 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.169226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130728.log 2023-12-24T21:18:50.659 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169226.log.gz 2023-12-24T21:18:50.659 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152069.log 2023-12-24T21:18:50.659 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137627.log: 56.2% -- replaced with /var/log/ceph/ceph-client.admin.137627.log.gz 2023-12-24T21:18:50.659 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121943.log 2023-12-24T21:18:50.659 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130728.log.gz 2023-12-24T21:18:50.660 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152069.log: gzip -5 --verbose 0.0%-- /var/log/ceph/ceph-client.admin.136327.log -- replaced with /var/log/ceph/ceph-client.admin.152069.log.gz 2023-12-24T21:18:50.660 INFO:teuthology.orchestra.run.smithi087.stderr: 2023-12-24T21:18:50.660 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142430.log 2023-12-24T21:18:50.660 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121943.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121943.log.gz 2023-12-24T21:18:50.660 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158432.log 2023-12-24T21:18:50.660 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136327.log.gz 2023-12-24T21:18:50.660 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142430.log: 1.2%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.142430.log.gz--verbose 2023-12-24T21:18:50.660 INFO:teuthology.orchestra.run.smithi087.stderr:-- /var/log/ceph/ceph-client.admin.144220.log 2023-12-24T21:18:50.660 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138759.log 2023-12-24T21:18:50.660 INFO:teuthology.orchestra.run.smithi087.stderr: 57.0% -- replaced with /var/log/ceph/ceph-client.admin.158432.log.gz 2023-12-24T21:18:50.661 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144220.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144220.log.gz 2023-12-24T21:18:50.661 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129213.log 2023-12-24T21:18:50.661 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.138759.log: -- /var/log/ceph/ceph-client.admin.128353.log 2023-12-24T21:18:50.661 INFO:teuthology.orchestra.run.smithi087.stderr: 53.9% -- replaced with /var/log/ceph/ceph-client.admin.138759.log.gz 2023-12-24T21:18:50.661 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145050.log 2023-12-24T21:18:50.661 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129213.log.gz 2023-12-24T21:18:50.661 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134975.log 2023-12-24T21:18:50.661 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128353.log.gz 2023-12-24T21:18:50.661 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136575.log 0.0% 2023-12-24T21:18:50.661 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.145050.log.gz 2023-12-24T21:18:50.662 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127843.log 2023-12-24T21:18:50.662 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134975.log.gz 2023-12-24T21:18:50.662 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127690.log 2023-12-24T21:18:50.662 INFO:teuthology.orchestra.run.smithi087.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.136575.log.gz 2023-12-24T21:18:50.662 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155725.log 2023-12-24T21:18:50.662 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127843.log.gz 2023-12-24T21:18:50.662 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123168.log 2023-12-24T21:18:50.662 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127690.log.gz 2023-12-24T21:18:50.662 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144757.log 2023-12-24T21:18:50.663 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155725.log.gz 2023-12-24T21:18:50.663 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123168.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.123168.log.gz--verbose 2023-12-24T21:18:50.663 INFO:teuthology.orchestra.run.smithi087.stderr:-- /var/log/ceph/ceph-client.admin.134013.log 2023-12-24T21:18:50.663 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159559.log 2023-12-24T21:18:50.663 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144757.log.gz 2023-12-24T21:18:50.663 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164106.log 2023-12-24T21:18:50.663 INFO:teuthology.orchestra.run.smithi087.stderr: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.134013.log.gz 2023-12-24T21:18:50.663 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163697.log 2023-12-24T21:18:50.663 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159559.log.gz 2023-12-24T21:18:50.663 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.164106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155644.log 2023-12-24T21:18:50.664 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164106.log.gz 2023-12-24T21:18:50.664 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170370.log 2023-12-24T21:18:50.664 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163697.log.gz 2023-12-24T21:18:50.664 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141621.log 2023-12-24T21:18:50.664 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155644.log.gz 2023-12-24T21:18:50.664 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153390.log 2023-12-24T21:18:50.664 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.170370.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170370.log.gz 2023-12-24T21:18:50.664 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140170.log 2023-12-24T21:18:50.664 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141621.log.gz 2023-12-24T21:18:50.664 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131476.log 2023-12-24T21:18:50.665 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153390.log.gz 2023-12-24T21:18:50.665 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163783.log 2023-12-24T21:18:50.665 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140170.log.gz 2023-12-24T21:18:50.665 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169978.log 2023-12-24T21:18:50.665 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131476.log.gz 2023-12-24T21:18:50.665 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163783.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.163783.log.gz-5 2023-12-24T21:18:50.665 INFO:teuthology.orchestra.run.smithi087.stderr:--verbose -- /var/log/ceph/ceph-client.admin.150941.log 2023-12-24T21:18:50.665 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.169978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140624.log 2023-12-24T21:18:50.665 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169978.log.gz 2023-12-24T21:18:50.666 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129093.log 2023-12-24T21:18:50.666 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150941.log.gz 2023-12-24T21:18:50.666 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144542.log 2023-12-24T21:18:50.666 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140624.log.gz 2023-12-24T21:18:50.666 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161421.log 2023-12-24T21:18:50.666 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129093.log.gz 2023-12-24T21:18:50.666 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158745.log 2023-12-24T21:18:50.666 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144542.log.gz 2023-12-24T21:18:50.666 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125473.log 2023-12-24T21:18:50.666 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161421.log.gz 2023-12-24T21:18:50.667 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160647.log 2023-12-24T21:18:50.667 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158745.log.gz 2023-12-24T21:18:50.667 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130592.log 2023-12-24T21:18:50.667 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125473.log.gz 2023-12-24T21:18:50.667 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129453.log 2023-12-24T21:18:50.667 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160647.log.gz 2023-12-24T21:18:50.667 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167034.log 2023-12-24T21:18:50.667 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130592.log.gz 2023-12-24T21:18:50.667 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159087.log 2023-12-24T21:18:50.667 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129453.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129453.log.gz 2023-12-24T21:18:50.668 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.167034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149815.log 2023-12-24T21:18:50.668 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167034.log.gz 2023-12-24T21:18:50.668 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156113.log 2023-12-24T21:18:50.668 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159087.log.gz 2023-12-24T21:18:50.668 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160091.log 2023-12-24T21:18:50.668 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149815.log.gz 2023-12-24T21:18:50.668 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128873.log 2023-12-24T21:18:50.668 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156113.log.gz 2023-12-24T21:18:50.668 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145926.log 2023-12-24T21:18:50.669 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160091.log.gz 2023-12-24T21:18:50.669 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135666.log 2023-12-24T21:18:50.669 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128873.log.gz 2023-12-24T21:18:50.669 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121047.log 2023-12-24T21:18:50.669 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145926.log.gz 2023-12-24T21:18:50.669 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128813.log 2023-12-24T21:18:50.669 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135666.log.gz 2023-12-24T21:18:50.669 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119184.log 2023-12-24T21:18:50.669 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121047.log.gz 2023-12-24T21:18:50.669 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132648.log 2023-12-24T21:18:50.670 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128813.log.gz 2023-12-24T21:18:50.670 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.119184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151940.log 2023-12-24T21:18:50.670 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119184.log.gz 2023-12-24T21:18:50.670 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132648.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.132648.log.gz-5 2023-12-24T21:18:50.670 INFO:teuthology.orchestra.run.smithi087.stderr:--verbose -- /var/log/ceph/ceph-client.admin.149372.log 2023-12-24T21:18:50.670 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157586.log 2023-12-24T21:18:50.670 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151940.log.gz 2023-12-24T21:18:50.670 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149372.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149372.log.gz 2023-12-24T21:18:50.670 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122888.log 2023-12-24T21:18:50.670 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151510.log 2023-12-24T21:18:50.671 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157586.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157586.log.gz 2023-12-24T21:18:50.671 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119822.log 2023-12-24T21:18:50.671 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122888.log.gz 2023-12-24T21:18:50.671 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154057.log 2023-12-24T21:18:50.671 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151510.log.gz 2023-12-24T21:18:50.671 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.119822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166464.log 2023-12-24T21:18:50.671 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119822.log.gz 2023-12-24T21:18:50.671 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154057.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.154057.log.gz--verbose 2023-12-24T21:18:50.671 INFO:teuthology.orchestra.run.smithi087.stderr:-- /var/log/ceph/ceph-client.admin.130373.log 2023-12-24T21:18:50.671 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130949.log 2023-12-24T21:18:50.672 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.166464.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166464.log.gz 2023-12-24T21:18:50.672 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148188.log 2023-12-24T21:18:50.672 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130373.log.gz 2023-12-24T21:18:50.672 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130949.log: 0.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.130949.log.gz--verbose 2023-12-24T21:18:50.672 INFO:teuthology.orchestra.run.smithi087.stderr:-- /var/log/ceph/ceph-client.admin.132375.log 2023-12-24T21:18:50.672 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146074.log 2023-12-24T21:18:50.672 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148188.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148188.log.gz 2023-12-24T21:18:50.672 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143681.log 2023-12-24T21:18:50.672 INFO:teuthology.orchestra.run.smithi087.stderr: 23.4% -- replaced with /var/log/ceph/ceph-client.admin.132375.log.gz 2023-12-24T21:18:50.673 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162311.log 2023-12-24T21:18:50.673 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146074.log.gz 2023-12-24T21:18:50.673 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140948.log 2023-12-24T21:18:50.673 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143681.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143681.log.gz 2023-12-24T21:18:50.673 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138002.log 2023-12-24T21:18:50.673 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162311.log.gz 2023-12-24T21:18:50.673 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151962.log 2023-12-24T21:18:50.673 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140948.log.gz 2023-12-24T21:18:50.673 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122037.log 2023-12-24T21:18:50.673 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138002.log: 23.1% -- replaced with /var/log/ceph/ceph-client.admin.138002.log.gz 2023-12-24T21:18:50.674 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156896.log 2023-12-24T21:18:50.674 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151962.log.gz 2023-12-24T21:18:50.674 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122037.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122037.log.gz 2023-12-24T21:18:50.674 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140342.log 2023-12-24T21:18:50.674 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151377.log 2023-12-24T21:18:50.674 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156896.log.gz 2023-12-24T21:18:50.674 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133383.log 2023-12-24T21:18:50.674 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140342.log.gz 2023-12-24T21:18:50.674 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151377.log: gzip -5 --verbose 0.0%-- /var/log/ceph/ceph-client.admin.162877.log -- replaced with /var/log/ceph/ceph-client.admin.151377.log.gz 2023-12-24T21:18:50.674 INFO:teuthology.orchestra.run.smithi087.stderr: 2023-12-24T21:18:50.675 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133383.log: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.133383.log.gz 2023-12-24T21:18:50.675 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162877.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162877.log.gz 2023-12-24T21:18:50.675 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134438.log 2023-12-24T21:18:50.675 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146114.log 2023-12-24T21:18:50.675 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163719.log 2023-12-24T21:18:50.675 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134438.log.gz 2023-12-24T21:18:50.675 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121413.log 2023-12-24T21:18:50.675 INFO:teuthology.orchestra.run.smithi087.stderr: 57.2% -- replaced with /var/log/ceph/ceph-client.admin.146114.log.gz 2023-12-24T21:18:50.675 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138023.log 2023-12-24T21:18:50.676 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163719.log.gz 2023-12-24T21:18:50.676 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121413.log: gzip -5 0.0%--verbose -- replaced with /var/log/ceph/ceph-client.admin.121413.log.gz-- 2023-12-24T21:18:50.676 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157469.log 2023-12-24T21:18:50.676 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129833.log 2023-12-24T21:18:50.676 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.138023.log.gz 2023-12-24T21:18:50.676 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130133.log 2023-12-24T21:18:50.676 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157469.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157469.log.gz 2023-12-24T21:18:50.676 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132711.log 2023-12-24T21:18:50.676 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129833.log.gz 2023-12-24T21:18:50.677 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149565.log 2023-12-24T21:18:50.677 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130133.log.gz 2023-12-24T21:18:50.677 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164063.log 2023-12-24T21:18:50.677 INFO:teuthology.orchestra.run.smithi087.stderr: 22.9% -- replaced with /var/log/ceph/ceph-client.admin.132711.log.gz 2023-12-24T21:18:50.677 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154788.log 2023-12-24T21:18:50.677 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149565.log.gz 2023-12-24T21:18:50.677 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130796.log 2023-12-24T21:18:50.677 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.164063.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164063.log.gz 2023-12-24T21:18:50.677 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145582.log 2023-12-24T21:18:50.677 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154788.log.gz 2023-12-24T21:18:50.678 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123649.log 2023-12-24T21:18:50.678 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130796.log.gz 2023-12-24T21:18:50.678 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152816.log 2023-12-24T21:18:50.678 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145582.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145582.log.gz 2023-12-24T21:18:50.678 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141535.log 2023-12-24T21:18:50.678 INFO:teuthology.orchestra.run.smithi087.stderr: 6.3% -- replaced with /var/log/ceph/ceph-client.admin.123649.log.gz 2023-12-24T21:18:50.678 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152816.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152816.log.gz 2023-12-24T21:18:50.678 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150921.log 2023-12-24T21:18:50.678 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160453.log 2023-12-24T21:18:50.678 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141535.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141535.log.gz 2023-12-24T21:18:50.679 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137526.log 2023-12-24T21:18:50.679 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150921.log.gz 2023-12-24T21:18:50.679 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158135.log 2023-12-24T21:18:50.679 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160453.log.gz 2023-12-24T21:18:50.679 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138361.log 2023-12-24T21:18:50.679 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137526.log.gz 2023-12-24T21:18:50.679 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167475.log 2023-12-24T21:18:50.679 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158135.log.gz 2023-12-24T21:18:50.679 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135267.log 2023-12-24T21:18:50.679 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.138361.log.gz 2023-12-24T21:18:50.680 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147567.log 2023-12-24T21:18:50.680 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.167475.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167475.log.gz 2023-12-24T21:18:50.680 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153433.log 2023-12-24T21:18:50.680 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135267.log.gz 2023-12-24T21:18:50.680 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147567.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147567.log.gz 2023-12-24T21:18:50.680 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123447.log 2023-12-24T21:18:50.680 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166887.log 2023-12-24T21:18:50.680 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153433.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153433.log.gz 2023-12-24T21:18:50.680 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123447.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.123447.log.gz-5 2023-12-24T21:18:50.681 INFO:teuthology.orchestra.run.smithi087.stderr:--verbose -- /var/log/ceph/ceph-client.admin.124154.log 2023-12-24T21:18:50.681 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164443.log 2023-12-24T21:18:50.681 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.166887.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166887.log.gz 2023-12-24T21:18:50.681 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135859.log 2023-12-24T21:18:50.681 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124154.log.gz 2023-12-24T21:18:50.681 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.164443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120309.log 2023-12-24T21:18:50.681 INFO:teuthology.orchestra.run.smithi087.stderr: 57.1% -- replaced with /var/log/ceph/ceph-client.admin.164443.log.gz 2023-12-24T21:18:50.681 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129053.log 2023-12-24T21:18:50.681 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135859.log.gz 2023-12-24T21:18:50.681 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.120309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148425.log 2023-12-24T21:18:50.682 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120309.log.gz 2023-12-24T21:18:50.682 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135133.log 2023-12-24T21:18:50.682 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129053.log.gz 2023-12-24T21:18:50.682 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121477.log 2023-12-24T21:18:50.682 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148425.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148425.log.gz 2023-12-24T21:18:50.682 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142207.log 2023-12-24T21:18:50.682 INFO:teuthology.orchestra.run.smithi087.stderr: 22.3% -- replaced with /var/log/ceph/ceph-client.admin.135133.log.gz 2023-12-24T21:18:50.682 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134266.log 2023-12-24T21:18:50.682 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121477.log.gz 2023-12-24T21:18:50.682 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135327.log 2023-12-24T21:18:50.683 INFO:teuthology.orchestra.run.smithi087.stderr: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.142207.log.gz 2023-12-24T21:18:50.683 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122178.log 2023-12-24T21:18:50.683 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134266.log.gz 2023-12-24T21:18:50.683 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135327.log: gzip 0.0%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.135327.log.gz-- 2023-12-24T21:18:50.683 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125336.log 2023-12-24T21:18:50.683 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134911.log 2023-12-24T21:18:50.683 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122178.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.122178.log.gz 2023-12-24T21:18:50.683 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150451.log 2023-12-24T21:18:50.683 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125336.log.gz 2023-12-24T21:18:50.684 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156073.log 2023-12-24T21:18:50.684 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134911.log.gz 2023-12-24T21:18:50.684 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127061.log 2023-12-24T21:18:50.684 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150451.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150451.log.gz 2023-12-24T21:18:50.684 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152404.log 2023-12-24T21:18:50.684 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156073.log.gz 2023-12-24T21:18:50.684 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135687.log 2023-12-24T21:18:50.684 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127061.log.gz 2023-12-24T21:18:50.684 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157336.log 2023-12-24T21:18:50.684 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152404.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152404.log.gz 2023-12-24T21:18:50.685 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155975.log 2023-12-24T21:18:50.685 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135687.log.gz 2023-12-24T21:18:50.685 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151323.log 2023-12-24T21:18:50.685 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157336.log.gz 2023-12-24T21:18:50.685 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146478.log 2023-12-24T21:18:50.685 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155975.log: 53.5% -- replaced with /var/log/ceph/ceph-client.admin.155975.log.gz 2023-12-24T21:18:50.685 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130541.log 2023-12-24T21:18:50.685 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151323.log.gz 2023-12-24T21:18:50.685 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162641.log 2023-12-24T21:18:50.685 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146478.log.gz 2023-12-24T21:18:50.686 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132543.log 2023-12-24T21:18:50.686 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130541.log.gz 2023-12-24T21:18:50.686 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125438.log 2023-12-24T21:18:50.686 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162641.log.gz 2023-12-24T21:18:50.686 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121305.log 2023-12-24T21:18:50.686 INFO:teuthology.orchestra.run.smithi087.stderr: 23.4% -- replaced with /var/log/ceph/ceph-client.admin.132543.log.gz 2023-12-24T21:18:50.686 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159345.log 2023-12-24T21:18:50.686 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125438.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.125438.log.gz 2023-12-24T21:18:50.686 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160926.log 2023-12-24T21:18:50.687 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121305.log.gz 2023-12-24T21:18:50.687 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160905.log 2023-12-24T21:18:50.687 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159345.log.gz 2023-12-24T21:18:50.687 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154487.log 2023-12-24T21:18:50.687 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160926.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160926.log.gz 2023-12-24T21:18:50.687 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147652.log 2023-12-24T21:18:50.687 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160905.log.gz 2023-12-24T21:18:50.687 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154487.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154487.log.gz 2023-12-24T21:18:50.687 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126026.log 2023-12-24T21:18:50.687 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129813.log 2023-12-24T21:18:50.688 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147652.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147652.log.gz 2023-12-24T21:18:50.688 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156799.log 2023-12-24T21:18:50.688 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126026.log.gz 2023-12-24T21:18:50.688 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127945.log 2023-12-24T21:18:50.688 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129813.log.gz 2023-12-24T21:18:50.688 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143832.log 2023-12-24T21:18:50.688 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156799.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156799.log.gz 2023-12-24T21:18:50.688 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129313.log 2023-12-24T21:18:50.688 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127945.log.gz 2023-12-24T21:18:50.689 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122845.log 2023-12-24T21:18:50.689 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143832.log.gz 2023-12-24T21:18:50.689 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152091.log 2023-12-24T21:18:50.689 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129313.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129313.log.gz 2023-12-24T21:18:50.689 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138580.log 2023-12-24T21:18:50.689 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122845.log.gz 2023-12-24T21:18:50.689 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137403.log 2023-12-24T21:18:50.689 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152091.log.gz 2023-12-24T21:18:50.689 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127707.log 2023-12-24T21:18:50.690 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138580.log.gz 2023-12-24T21:18:50.690 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153222.log 2023-12-24T21:18:50.690 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137403.log.gz 2023-12-24T21:18:50.690 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127758.log 2023-12-24T21:18:50.690 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127707.log.gz 2023-12-24T21:18:50.690 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123232.log 2023-12-24T21:18:50.690 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153222.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153222.log.gz 2023-12-24T21:18:50.690 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156916.log 2023-12-24T21:18:50.690 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127758.log.gz 2023-12-24T21:18:50.690 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162598.log 2023-12-24T21:18:50.691 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123232.log.gz 2023-12-24T21:18:50.691 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-12-24T21:18:50.691 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156916.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156916.log.gz 2023-12-24T21:18:50.691 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128389.log 2023-12-24T21:18:50.691 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162598.log.gz 2023-12-24T21:18:50.691 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130813.log 2023-12-24T21:18:50.691 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128389.log: 56.4% -- replaced with /var/log/ceph/ceph-client.admin.128389.log.gz 2023-12-24T21:18:50.691 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144284.log 2023-12-24T21:18:50.691 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130813.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130813.log.gz 2023-12-24T21:18:50.691 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163250.log 2023-12-24T21:18:50.692 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124090.log 2023-12-24T21:18:50.692 INFO:teuthology.orchestra.run.smithi087.stderr: 26.0% -- replaced with /var/log/ceph/ceph-client.admin.144284.log.gz 2023-12-24T21:18:50.692 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162619.log 2023-12-24T21:18:50.692 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163250.log.gz 2023-12-24T21:18:50.692 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124090.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124090.log.gz 2023-12-24T21:18:50.692 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162619.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162619.log.gz 2023-12-24T21:18:50.692 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150344.log 2023-12-24T21:18:50.692 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142735.log 2023-12-24T21:18:50.692 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150344.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150344.log.gz 2023-12-24T21:18:50.692 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142735.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142735.log.gz 2023-12-24T21:18:50.693 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145475.log 2023-12-24T21:18:50.693 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126241.log 2023-12-24T21:18:50.693 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126874.log 2023-12-24T21:18:50.693 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145475.log.gz 2023-12-24T21:18:50.693 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126241.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126241.log.gz 2023-12-24T21:18:50.693 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126874.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.126874.log.gz 2023-12-24T21:18:50.693 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138102.log 2023-12-24T21:18:50.693 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169683.log 2023-12-24T21:18:50.693 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147098.log 2023-12-24T21:18:50.694 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138102.log.gz 2023-12-24T21:18:50.694 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.169683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141793.log 2023-12-24T21:18:50.694 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169683.log.gz 2023-12-24T21:18:50.694 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147098.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147098.log.gz 2023-12-24T21:18:50.694 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141793.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141793.log.gz 2023-12-24T21:18:50.694 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134631.log 2023-12-24T21:18:50.694 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122538.log 2023-12-24T21:18:50.694 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134631.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134631.log.gz 2023-12-24T21:18:50.694 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122538.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122538.log.gz 2023-12-24T21:18:50.695 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138701.log 2023-12-24T21:18:50.695 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137775.log 2023-12-24T21:18:50.695 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171472.log 2023-12-24T21:18:50.695 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138701.log.gz 2023-12-24T21:18:50.695 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146758.log 2023-12-24T21:18:50.695 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137775.log.gz 2023-12-24T21:18:50.695 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.171472.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171472.log.gz 2023-12-24T21:18:50.695 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146758.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146758.log.gz 2023-12-24T21:18:50.695 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148146.log 2023-12-24T21:18:50.695 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157256.log 2023-12-24T21:18:50.696 INFO:teuthology.orchestra.run.smithi087.stderr:gzip /var/log/ceph/ceph-client.admin.148146.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.161077.log 2023-12-24T21:18:50.696 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148146.log.gz 2023-12-24T21:18:50.696 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157256.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157256.log.gz 2023-12-24T21:18:50.696 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161077.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161077.log.gz 2023-12-24T21:18:50.696 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161464.log 2023-12-24T21:18:50.696 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139565.log 2023-12-24T21:18:50.696 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161464.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161464.log.gz 2023-12-24T21:18:50.696 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139565.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139565.log.gz 2023-12-24T21:18:50.696 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131340.log 2023-12-24T21:18:50.697 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131663.log 2023-12-24T21:18:50.697 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151043.log 2023-12-24T21:18:50.697 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131340.log.gz 2023-12-24T21:18:50.697 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139042.log 2023-12-24T21:18:50.697 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131663.log.gz 2023-12-24T21:18:50.697 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151043.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151043.log.gz 2023-12-24T21:18:50.697 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139042.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139042.log.gz 2023-12-24T21:18:50.697 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160174.log 2023-12-24T21:18:50.697 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149243.log 2023-12-24T21:18:50.698 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134118.log 2023-12-24T21:18:50.698 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160174.log.gz 2023-12-24T21:18:50.698 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149243.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149243.log.gz 2023-12-24T21:18:50.698 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134118.log: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.134118.log.gz 2023-12-24T21:18:50.698 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145969.log 2023-12-24T21:18:50.698 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163891.log 2023-12-24T21:18:50.698 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145969.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145969.log.gz 2023-12-24T21:18:50.698 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163891.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163891.log.gz 2023-12-24T21:18:50.698 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131867.log 2023-12-24T21:18:50.698 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154766.log 2023-12-24T21:18:50.699 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142571.log 2023-12-24T21:18:50.699 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131867.log.gz 2023-12-24T21:18:50.699 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154766.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154766.log.gz 2023-12-24T21:18:50.699 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142571.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142571.log.gz 2023-12-24T21:18:50.699 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145754.log 2023-12-24T21:18:50.699 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130898.log 2023-12-24T21:18:50.699 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171374.log 2023-12-24T21:18:50.699 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145754.log.gz 2023-12-24T21:18:50.699 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130898.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130898.log.gz 2023-12-24T21:18:50.700 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.171374.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171374.log.gz 2023-12-24T21:18:50.700 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141407.log 2023-12-24T21:18:50.700 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124454.log 2023-12-24T21:18:50.700 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146779.log 2023-12-24T21:18:50.700 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141407.log.gz 2023-12-24T21:18:50.700 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157747.log 2023-12-24T21:18:50.700 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124454.log.gz 2023-12-24T21:18:50.700 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146779.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146779.log.gz 2023-12-24T21:18:50.700 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157747.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157747.log.gz 2023-12-24T21:18:50.700 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137317.log 2023-12-24T21:18:50.701 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136224.log 2023-12-24T21:18:50.701 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137317.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137317.log.gz 2023-12-24T21:18:50.701 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136224.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136224.log.gz 2023-12-24T21:18:50.701 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124820.log 2023-12-24T21:18:50.701 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124540.log 2023-12-24T21:18:50.701 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135816.log 2023-12-24T21:18:50.701 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124820.log.gz 2023-12-24T21:18:50.701 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124540.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124540.log.gz 2023-12-24T21:18:50.701 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135816.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135816.log.gz 2023-12-24T21:18:50.702 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129153.log 2023-12-24T21:18:50.702 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131442.log 2023-12-24T21:18:50.702 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171725.log 2023-12-24T21:18:50.702 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129153.log.gz 2023-12-24T21:18:50.702 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131442.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131442.log.gz 2023-12-24T21:18:50.702 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.171725.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171725.log.gz 2023-12-24T21:18:50.702 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149920.log 2023-12-24T21:18:50.702 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171823.log 2023-12-24T21:18:50.702 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126721.log 2023-12-24T21:18:50.702 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149920.log.gz 2023-12-24T21:18:50.703 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.171823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154336.log 2023-12-24T21:18:50.703 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171823.log.gz 2023-12-24T21:18:50.703 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126721.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.126721.log.gz 2023-12-24T21:18:50.703 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154336.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154336.log.gz 2023-12-24T21:18:50.703 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169373.log 2023-12-24T21:18:50.703 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170027.log 2023-12-24T21:18:50.703 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130830.log 2023-12-24T21:18:50.703 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.169373.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169373.log.gz 2023-12-24T21:18:50.703 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.170027.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170027.log.gz 2023-12-24T21:18:50.704 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130830.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130830.log.gz 2023-12-24T21:18:50.704 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160690.log 2023-12-24T21:18:50.704 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124047.log 2023-12-24T21:18:50.704 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160690.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160690.log.gz 2023-12-24T21:18:50.704 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124047.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124047.log.gz 2023-12-24T21:18:50.704 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159237.log 2023-12-24T21:18:50.704 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143596.log 2023-12-24T21:18:50.704 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128593.log 2023-12-24T21:18:50.704 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159237.log.gz 2023-12-24T21:18:50.704 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143596.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143596.log.gz 2023-12-24T21:18:50.705 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128593.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128593.log.gz 2023-12-24T21:18:50.705 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129793.log 2023-12-24T21:18:50.705 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126755.log 2023-12-24T21:18:50.705 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129793.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129793.log.gz 2023-12-24T21:18:50.705 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126755.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.126755.log.gz 2023-12-24T21:18:50.705 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146010.log 2023-12-24T21:18:50.705 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163590.log 2023-12-24T21:18:50.705 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161356.log 2023-12-24T21:18:50.705 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146010.log.gz 2023-12-24T21:18:50.706 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120654.log 2023-12-24T21:18:50.706 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163590.log.gz 2023-12-24T21:18:50.706 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161356.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161356.log.gz 2023-12-24T21:18:50.706 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.120654.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120654.log.gz 2023-12-24T21:18:50.706 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131170.log 2023-12-24T21:18:50.706 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130983.log 2023-12-24T21:18:50.706 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131170.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131170.log.gz 2023-12-24T21:18:50.706 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130983.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130983.log.gz 2023-12-24T21:18:50.706 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142901.log 2023-12-24T21:18:50.706 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127673.log 2023-12-24T21:18:50.707 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152505.log 2023-12-24T21:18:50.707 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142901.log.gz 2023-12-24T21:18:50.707 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133572.log 2023-12-24T21:18:50.707 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127673.log.gz 2023-12-24T21:18:50.707 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152505.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152505.log.gz 2023-12-24T21:18:50.707 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133572.log: 23.5% -- replaced with /var/log/ceph/ceph-client.admin.133572.log.gz 2023-12-24T21:18:50.707 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160496.log 2023-12-24T21:18:50.707 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142137.log 2023-12-24T21:18:50.707 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136937.log 2023-12-24T21:18:50.707 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160496.log.gz 2023-12-24T21:18:50.708 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142137.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142137.log.gz 2023-12-24T21:18:50.708 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136937.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.136937.log.gz 2023-12-24T21:18:50.708 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140838.log 2023-12-24T21:18:50.708 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136737.log 2023-12-24T21:18:50.708 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140838.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140838.log.gz 2023-12-24T21:18:50.708 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136737.log: gzip -5 0.0%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.136737.log.gz/var/log/ceph/ceph-client.admin.166527.log 2023-12-24T21:18:50.708 INFO:teuthology.orchestra.run.smithi087.stderr: 2023-12-24T21:18:50.708 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134160.log 2023-12-24T21:18:50.708 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.166527.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166527.log.gz 2023-12-24T21:18:50.709 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134160.log: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.134160.log.gz 2023-12-24T21:18:50.709 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159216.log 2023-12-24T21:18:50.709 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162068.log 2023-12-24T21:18:50.709 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127894.log 2023-12-24T21:18:50.709 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159216.log.gz 2023-12-24T21:18:50.709 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162068.log: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.162068.log.gz 2023-12-24T21:18:50.709 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127894.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127894.log.gz 2023-12-24T21:18:50.709 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123629.log 2023-12-24T21:18:50.709 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154982.log 2023-12-24T21:18:50.709 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167981.log 2023-12-24T21:18:50.710 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123629.log.gz 2023-12-24T21:18:50.710 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154982.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154982.log.gz 2023-12-24T21:18:50.710 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.167981.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167981.log.gz 2023-12-24T21:18:50.710 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137587.log 2023-12-24T21:18:50.710 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145862.log 2023-12-24T21:18:50.710 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137587.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137587.log.gz 2023-12-24T21:18:50.710 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145862.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145862.log.gz 2023-12-24T21:18:50.710 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158766.log 2023-12-24T21:18:50.710 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151682.log 2023-12-24T21:18:50.710 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121905.log 2023-12-24T21:18:50.711 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158766.log.gz 2023-12-24T21:18:50.711 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125681.log 2023-12-24T21:18:50.711 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151682.log.gz 2023-12-24T21:18:50.711 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121905.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121905.log.gz 2023-12-24T21:18:50.711 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125681.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125681.log.gz 2023-12-24T21:18:50.711 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153347.log 2023-12-24T21:18:50.711 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127248.log 2023-12-24T21:18:50.711 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153347.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153347.log.gz 2023-12-24T21:18:50.711 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127248.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127248.log.gz 2023-12-24T21:18:50.712 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157863.log 2023-12-24T21:18:50.712 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145196.log 2023-12-24T21:18:50.712 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157863.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157863.log.gz 2023-12-24T21:18:50.712 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145196.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145196.log.gz 2023-12-24T21:18:50.712 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121799.log 2023-12-24T21:18:50.712 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125293.log 2023-12-24T21:18:50.712 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143290.log 2023-12-24T21:18:50.712 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121799.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121799.log.gz 2023-12-24T21:18:50.712 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153949.log 2023-12-24T21:18:50.712 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125293.log.gz 2023-12-24T21:18:50.713 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143290.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143290.log.gz 2023-12-24T21:18:50.713 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153949.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153949.log.gz 2023-12-24T21:18:50.713 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154508.log 2023-12-24T21:18:50.713 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154379.log 2023-12-24T21:18:50.713 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154508.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154508.log.gz 2023-12-24T21:18:50.713 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154379.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154379.log.gz 2023-12-24T21:18:50.713 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143403.log 2023-12-24T21:18:50.713 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126925.log 2023-12-24T21:18:50.713 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142051.log 2023-12-24T21:18:50.714 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143403.log.gz 2023-12-24T21:18:50.714 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126925.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.126925.log.gz 2023-12-24T21:18:50.714 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142051.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142051.log.gz 2023-12-24T21:18:50.714 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146034.log 2023-12-24T21:18:50.714 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130609.log 2023-12-24T21:18:50.714 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146034.log: 28.5% -- replaced with /var/log/ceph/ceph-client.admin.146034.log.gz 2023-12-24T21:18:50.714 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130609.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130609.log.gz 2023-12-24T21:18:50.714 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126857.log 2023-12-24T21:18:50.714 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143424.log 2023-12-24T21:18:50.714 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126857.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.126857.log.gz 2023-12-24T21:18:50.715 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143424.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143424.log.gz 2023-12-24T21:18:50.715 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157546.log 2023-12-24T21:18:50.715 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122464.log 2023-12-24T21:18:50.715 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157546.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157546.log.gz 2023-12-24T21:18:50.715 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122464.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122464.log.gz 2023-12-24T21:18:50.715 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137795.log 2023-12-24T21:18:50.715 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133698.log 2023-12-24T21:18:50.715 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125961.log 2023-12-24T21:18:50.715 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137795.log.gz 2023-12-24T21:18:50.715 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168471.log 2023-12-24T21:18:50.716 INFO:teuthology.orchestra.run.smithi087.stderr: 23.4% -- replaced with /var/log/ceph/ceph-client.admin.133698.log.gz 2023-12-24T21:18:50.716 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125961.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125961.log.gz 2023-12-24T21:18:50.716 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.168471.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168471.log.gz 2023-12-24T21:18:50.716 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161399.log 2023-12-24T21:18:50.716 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167328.log 2023-12-24T21:18:50.716 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127554.log 2023-12-24T21:18:50.716 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161399.log.gz 2023-12-24T21:18:50.716 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.167328.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167328.log.gz 2023-12-24T21:18:50.716 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127554.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.127554.log.gz 2023-12-24T21:18:50.716 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155254.log 2023-12-24T21:18:50.717 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124283.log 2023-12-24T21:18:50.717 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155254.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155254.log.gz 2023-12-24T21:18:50.717 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124283.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124283.log.gz 2023-12-24T21:18:50.717 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124219.log 2023-12-24T21:18:50.717 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150365.log 2023-12-24T21:18:50.717 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126772.log 2023-12-24T21:18:50.717 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124219.log.gz 2023-12-24T21:18:50.717 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150365.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150365.log.gz 2023-12-24T21:18:50.717 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126772.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.126772.log.gz 2023-12-24T21:18:50.718 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140063.log 2023-12-24T21:18:50.718 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129633.log 2023-12-24T21:18:50.718 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140063.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140063.log.gz 2023-12-24T21:18:50.718 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129633.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129633.log.gz 2023-12-24T21:18:50.718 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139059.log 2023-12-24T21:18:50.718 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138160.log 2023-12-24T21:18:50.718 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145111.log 2023-12-24T21:18:50.718 INFO:teuthology.orchestra.run.smithi087.stderr: 24.2% -- replaced with /var/log/ceph/ceph-client.admin.139059.log.gz 2023-12-24T21:18:50.718 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138160.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138160.log.gz 2023-12-24T21:18:50.718 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145111.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145111.log.gz 2023-12-24T21:18:50.719 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150215.log 2023-12-24T21:18:50.719 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140278.log 2023-12-24T21:18:50.719 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150215.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150215.log.gz 2023-12-24T21:18:50.719 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140278.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140278.log.gz 2023-12-24T21:18:50.719 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156876.log 2023-12-24T21:18:50.719 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127622.log 2023-12-24T21:18:50.719 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156876.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156876.log.gz 2023-12-24T21:18:50.719 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127622.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127622.log.gz 2023-12-24T21:18:50.719 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146629.log 2023-12-24T21:18:50.719 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153863.log 2023-12-24T21:18:50.720 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146629.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146629.log.gz 2023-12-24T21:18:50.720 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153863.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153863.log.gz 2023-12-24T21:18:50.720 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123490.log 2023-12-24T21:18:50.720 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130915.log 2023-12-24T21:18:50.720 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153691.log 2023-12-24T21:18:50.720 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123490.log.gz 2023-12-24T21:18:50.720 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132480.log 2023-12-24T21:18:50.720 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130915.log.gz 2023-12-24T21:18:50.720 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153691.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153691.log.gz 2023-12-24T21:18:50.720 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132480.log: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.132480.log.gz 2023-12-24T21:18:50.721 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122740.log 2023-12-24T21:18:50.721 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141643.log 2023-12-24T21:18:50.721 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128973.log 2023-12-24T21:18:50.721 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122740.log.gz 2023-12-24T21:18:50.721 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141643.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141643.log.gz 2023-12-24T21:18:50.721 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128973.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128973.log.gz 2023-12-24T21:18:50.721 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143638.log 2023-12-24T21:18:50.721 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124433.log 2023-12-24T21:18:50.721 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159538.log 2023-12-24T21:18:50.722 INFO:teuthology.orchestra.run.smithi087.stderr: 57.2% -- replaced with /var/log/ceph/ceph-client.admin.143638.log.gz 2023-12-24T21:18:50.722 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124433.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124433.log.gz 2023-12-24T21:18:50.722 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159538.log: 0.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.159538.log.gz--verbose 2023-12-24T21:18:50.722 INFO:teuthology.orchestra.run.smithi087.stderr:-- /var/log/ceph/ceph-client.admin.122614.log 2023-12-24T21:18:50.722 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126155.log 2023-12-24T21:18:50.722 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143810.log 2023-12-24T21:18:50.722 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122614.log.gz 2023-12-24T21:18:50.722 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124025.log 2023-12-24T21:18:50.722 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126155.log.gz 2023-12-24T21:18:50.722 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143810.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143810.log.gz 2023-12-24T21:18:50.723 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124025.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124025.log.gz 2023-12-24T21:18:50.723 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146822.log 2023-12-24T21:18:50.723 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156856.log 2023-12-24T21:18:50.723 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143337.log 2023-12-24T21:18:50.723 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146822.log.gz 2023-12-24T21:18:50.723 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156856.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156856.log.gz 2023-12-24T21:18:50.723 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143337.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143337.log.gz 2023-12-24T21:18:50.723 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129753.log 2023-12-24T21:18:50.723 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126284.log 2023-12-24T21:18:50.724 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129753.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129753.log.gz 2023-12-24T21:18:50.724 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126284.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126284.log.gz 2023-12-24T21:18:50.724 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154594.log 2023-12-24T21:18:50.724 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153713.log 2023-12-24T21:18:50.724 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144881.log 2023-12-24T21:18:50.724 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154594.log.gz 2023-12-24T21:18:50.724 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168275.log 2023-12-24T21:18:50.724 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153713.log.gz 2023-12-24T21:18:50.724 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144881.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144881.log.gz 2023-12-24T21:18:50.724 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.168275.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168275.log.gz 2023-12-24T21:18:50.725 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128285.log 2023-12-24T21:18:50.725 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122867.log 2023-12-24T21:18:50.725 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128285.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.128285.log.gz 2023-12-24T21:18:50.725 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122867.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122867.log.gz 2023-12-24T21:18:50.725 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158493.log 2023-12-24T21:18:50.725 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160006.log 2023-12-24T21:18:50.725 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158493.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158493.log.gz 2023-12-24T21:18:50.725 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160006.log: 52.9% -- replaced with /var/log/ceph/ceph-client.admin.160006.log.gz 2023-12-24T21:18:50.725 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138638.log 2023-12-24T21:18:50.726 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137444.log 2023-12-24T21:18:50.726 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133362.log 2023-12-24T21:18:50.726 INFO:teuthology.orchestra.run.smithi087.stderr: 23.4% -- replaced with /var/log/ceph/ceph-client.admin.138638.log.gz 2023-12-24T21:18:50.726 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137444.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137444.log.gz 2023-12-24T21:18:50.726 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133362.log: 23.1% -- replaced with /var/log/ceph/ceph-client.admin.133362.log.gz 2023-12-24T21:18:50.726 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155434.log 2023-12-24T21:18:50.726 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170476.log 2023-12-24T21:18:50.726 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155434.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155434.log.gz 2023-12-24T21:18:50.726 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.170476.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170476.log.gz 2023-12-24T21:18:50.726 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126823.log 2023-12-24T21:18:50.727 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132228.log 2023-12-24T21:18:50.727 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136268.log 2023-12-24T21:18:50.727 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.126823.log.gz 2023-12-24T21:18:50.727 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158453.log 2023-12-24T21:18:50.727 INFO:teuthology.orchestra.run.smithi087.stderr: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.132228.log.gz 2023-12-24T21:18:50.727 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161550.log 2023-12-24T21:18:50.727 INFO:teuthology.orchestra.run.smithi087.stderr: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.136268.log.gz 2023-12-24T21:18:50.727 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131153.log 2023-12-24T21:18:50.727 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158453.log.gz 2023-12-24T21:18:50.728 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161550.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161550.log.gz 2023-12-24T21:18:50.728 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131153.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131153.log.gz 2023-12-24T21:18:50.728 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154849.log 2023-12-24T21:18:50.728 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141428.log 2023-12-24T21:18:50.728 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122122.log 2023-12-24T21:18:50.728 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154849.log.gz 2023-12-24T21:18:50.728 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141428.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141428.log.gz 2023-12-24T21:18:50.728 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122122.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122122.log.gz 2023-12-24T21:18:50.728 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153562.log 2023-12-24T21:18:50.728 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149436.log 2023-12-24T21:18:50.729 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160260.log 2023-12-24T21:18:50.729 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153562.log.gz 2023-12-24T21:18:50.729 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149436.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149436.log.gz 2023-12-24T21:18:50.729 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160260.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160260.log.gz 2023-12-24T21:18:50.729 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139866.log 2023-12-24T21:18:50.729 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164127.log 2023-12-24T21:18:50.729 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161335.log 2023-12-24T21:18:50.729 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139866.log.gz 2023-12-24T21:18:50.729 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.164127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155374.log 2023-12-24T21:18:50.730 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164127.log.gz 2023-12-24T21:18:50.730 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161335.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161335.log.gz 2023-12-24T21:18:50.730 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155374.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155374.log.gz 2023-12-24T21:18:50.730 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136636.log 2023-12-24T21:18:50.730 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119525.log 2023-12-24T21:18:50.730 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136636.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136636.log.gz 2023-12-24T21:18:50.730 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.119525.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119525.log.gz 2023-12-24T21:18:50.730 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163955.log 2023-12-24T21:18:50.730 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154702.log 2023-12-24T21:18:50.730 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152527.log 2023-12-24T21:18:50.731 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163955.log.gz 2023-12-24T21:18:50.731 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122576.log 2023-12-24T21:18:50.731 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154702.log.gz 2023-12-24T21:18:50.731 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152527.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152527.log.gz 2023-12-24T21:18:50.731 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122576.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122576.log.gz 2023-12-24T21:18:50.731 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159173.log 2023-12-24T21:18:50.731 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161378.log 2023-12-24T21:18:50.731 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132071.log 2023-12-24T21:18:50.731 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159173.log.gz 2023-12-24T21:18:50.732 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161378.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161378.log.gz 2023-12-24T21:18:50.732 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132071.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132071.log.gz 2023-12-24T21:18:50.732 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121563.log 2023-12-24T21:18:50.732 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162271.log 2023-12-24T21:18:50.732 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123297.log 2023-12-24T21:18:50.732 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121563.log.gz 2023-12-24T21:18:50.732 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162271.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162271.log.gz 2023-12-24T21:18:50.732 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123297.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123297.log.gz 2023-12-24T21:18:50.732 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136554.log 2023-12-24T21:18:50.732 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160883.log 2023-12-24T21:18:50.733 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126942.log 2023-12-24T21:18:50.733 INFO:teuthology.orchestra.run.smithi087.stderr: 57.2% -- replaced with /var/log/ceph/ceph-client.admin.136554.log.gz 2023-12-24T21:18:50.733 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132690.log 2023-12-24T21:18:50.733 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160883.log.gz 2023-12-24T21:18:50.733 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126942.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.126942.log.gz 2023-12-24T21:18:50.733 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132690.log: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.132690.log.gz 2023-12-24T21:18:50.733 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152048.log 2023-12-24T21:18:50.822 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130398.log 2023-12-24T21:18:50.822 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152048.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152048.log.gz 2023-12-24T21:18:50.822 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130398.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130398.log.gz 2023-12-24T21:18:50.822 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152026.log 2023-12-24T21:18:50.823 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160561.log 2023-12-24T21:18:50.823 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152026.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-24T21:18:50.823 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152026.log.gz 2023-12-24T21:18:50.823 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141578.log 2023-12-24T21:18:50.823 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160561.log.gz 2023-12-24T21:18:50.823 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.141578.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141578.log.gz 2023-12-24T21:18:50.823 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134502.log 2023-12-24T21:18:50.823 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172525.log 2023-12-24T21:18:50.823 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134502.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134502.log.gz 2023-12-24T21:18:50.823 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.172525.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.172525.log.gz 2023-12-24T21:18:50.824 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135116.log 2023-12-24T21:18:50.824 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158807.log 2023-12-24T21:18:50.824 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135116.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135116.log.gz 2023-12-24T21:18:50.824 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158807.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158807.log.gz 2023-12-24T21:18:50.824 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127384.log 2023-12-24T21:18:50.824 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127656.log 2023-12-24T21:18:50.824 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127384.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127384.log.gz 2023-12-24T21:18:50.824 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127656.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127656.log.gz 2023-12-24T21:18:50.824 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123769.log 2023-12-24T21:18:50.825 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145668.log 2023-12-24T21:18:50.825 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123769.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123769.log.gz 2023-12-24T21:18:50.825 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145668.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145668.log.gz 2023-12-24T21:18:50.825 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127333.log 2023-12-24T21:18:50.825 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167132.log 2023-12-24T21:18:50.825 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127333.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127333.log.gz 2023-12-24T21:18:50.825 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.167132.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167132.log.gz 2023-12-24T21:18:50.825 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152112.log 2023-12-24T21:18:50.825 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140602.log 2023-12-24T21:18:50.826 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152112.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152112.log.gz 2023-12-24T21:18:50.826 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140602.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140602.log.gz 2023-12-24T21:18:50.826 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164020.log 2023-12-24T21:18:50.826 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145155.log 2023-12-24T21:18:50.826 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.164020.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164020.log.gz 2023-12-24T21:18:50.826 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145155.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145155.log.gz 2023-12-24T21:18:50.826 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162684.log 2023-12-24T21:18:50.826 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138979.log 2023-12-24T21:18:50.826 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162684.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162684.log.gz 2023-12-24T21:18:50.826 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138979.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138979.log.gz 2023-12-24T21:18:50.827 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140385.log 2023-12-24T21:18:50.827 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166625.log 2023-12-24T21:18:50.827 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140385.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140385.log.gz 2023-12-24T21:18:50.827 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.166625.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166625.log.gz 2023-12-24T21:18:50.827 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159495.log 2023-12-24T21:18:50.827 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129913.log 2023-12-24T21:18:50.827 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159495.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159495.log.gz 2023-12-24T21:18:50.827 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129913.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129913.log.gz 2023-12-24T21:18:50.827 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130575.log 2023-12-24T21:18:50.827 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130575.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130575.log.gz 2023-12-24T21:18:50.828 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141314.log 2023-12-24T21:18:50.828 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139221.log 2023-12-24T21:18:50.828 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141314.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141314.log.gz 2023-12-24T21:18:50.828 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139221.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139221.log.gz 2023-12-24T21:18:50.828 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138518.log 2023-12-24T21:18:50.828 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121434.log 2023-12-24T21:18:50.828 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138518.log: 22.4% -- replaced with /var/log/ceph/ceph-client.admin.138518.log.gz 2023-12-24T21:18:50.828 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121434.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121434.log.gz 2023-12-24T21:18:50.828 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150430.log 2023-12-24T21:18:50.829 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125918.log 2023-12-24T21:18:50.829 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150430.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150430.log.gz 2023-12-24T21:18:50.829 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125918.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125918.log.gz 2023-12-24T21:18:50.829 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152601.log 2023-12-24T21:18:50.829 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134825.log 2023-12-24T21:18:50.829 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152601.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152601.log.gz 2023-12-24T21:18:50.829 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134825.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134825.log.gz 2023-12-24T21:18:50.829 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163335.log 2023-12-24T21:18:50.829 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154961.log 2023-12-24T21:18:50.830 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163335.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163335.log.gz 2023-12-24T21:18:50.830 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154961.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154961.log.gz 2023-12-24T21:18:50.830 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114581.log 2023-12-24T21:18:50.830 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.114581.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114581.log.gz 2023-12-24T21:18:50.830 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139328.log 2023-12-24T21:18:50.830 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122274.log 2023-12-24T21:18:50.830 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139328.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139328.log.gz 2023-12-24T21:18:50.830 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122274.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122274.log.gz 2023-12-24T21:18:50.830 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148920.log 2023-12-24T21:18:50.830 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140192.log 2023-12-24T21:18:50.831 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148920.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148920.log.gz 2023-12-24T21:18:50.831 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140192.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140192.log.gz 2023-12-24T21:18:50.831 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157606.log 2023-12-24T21:18:50.831 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148125.log 2023-12-24T21:18:50.831 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157606.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157606.log.gz 2023-12-24T21:18:50.831 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148125.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148125.log.gz 2023-12-24T21:18:50.831 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127911.log 2023-12-24T21:18:50.831 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126434.log 2023-12-24T21:18:50.831 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127911.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127911.log.gz 2023-12-24T21:18:50.831 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126434.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126434.log.gz 2023-12-24T21:18:50.831 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132291.log 2023-12-24T21:18:50.832 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161292.log 2023-12-24T21:18:50.832 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132291.log: 23.1% -- replaced with /var/log/ceph/ceph-client.admin.132291.log.gz 2023-12-24T21:18:50.832 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161292.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161292.log.gz 2023-12-24T21:18:50.832 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163229.log 2023-12-24T21:18:50.832 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131884.log 2023-12-24T21:18:50.832 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163229.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163229.log.gz 2023-12-24T21:18:50.832 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131884.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131884.log.gz 2023-12-24T21:18:50.832 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152776.log 2023-12-24T21:18:50.832 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123125.log 2023-12-24T21:18:50.833 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152776.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152776.log.gz 2023-12-24T21:18:50.833 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123125.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123125.log.gz 2023-12-24T21:18:50.833 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153820.log 2023-12-24T21:18:50.833 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122931.log 2023-12-24T21:18:50.833 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153820.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153820.log.gz 2023-12-24T21:18:50.833 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122931.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122931.log.gz 2023-12-24T21:18:50.833 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167181.log 2023-12-24T21:18:50.833 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162942.log 2023-12-24T21:18:50.833 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.167181.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167181.log.gz 2023-12-24T21:18:50.834 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162942.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162942.log.gz 2023-12-24T21:18:50.834 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157099.log 2023-12-24T21:18:50.834 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149479.log 2023-12-24T21:18:50.834 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157099.log: 55.2% -- replaced with /var/log/ceph/ceph-client.admin.157099.log.gz 2023-12-24T21:18:50.834 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149479.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149479.log.gz 2023-12-24T21:18:50.834 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162834.log 2023-12-24T21:18:50.834 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157120.log 2023-12-24T21:18:50.834 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162834.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162834.log.gz 2023-12-24T21:18:50.834 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157120.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157120.log.gz 2023-12-24T21:18:50.834 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141019.log 2023-12-24T21:18:50.835 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159259.log 2023-12-24T21:18:50.835 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141019.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141019.log.gz 2023-12-24T21:18:50.835 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159576.log 2023-12-24T21:18:50.835 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159259.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159259.log.gz 2023-12-24T21:18:50.835 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128513.log 2023-12-24T21:18:50.835 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159576.log.gz 2023-12-24T21:18:50.835 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159022.log 2023-12-24T21:18:50.835 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128513.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128513.log.gz 2023-12-24T21:18:50.835 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159022.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159022.log.gz 2023-12-24T21:18:50.835 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145346.log 2023-12-24T21:18:50.836 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122719.log 2023-12-24T21:18:50.836 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145346.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145346.log.gz 2023-12-24T21:18:50.836 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122719.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122719.log.gz 2023-12-24T21:18:50.836 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124884.log 2023-12-24T21:18:50.836 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140688.log 2023-12-24T21:18:50.836 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124884.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124884.log.gz 2023-12-24T21:18:50.836 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140688.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140688.log.gz 2023-12-24T21:18:50.836 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157059.log 2023-12-24T21:18:50.836 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141389.log 2023-12-24T21:18:50.836 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157059.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157059.log.gz 2023-12-24T21:18:50.837 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141389.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141389.log.gz 2023-12-24T21:18:50.837 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135429.log 2023-12-24T21:18:50.837 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157943.log 2023-12-24T21:18:50.837 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135429.log: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.135429.log.gz 2023-12-24T21:18:50.837 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134588.log 2023-12-24T21:18:50.837 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157943.log.gz 2023-12-24T21:18:50.837 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150582.log 2023-12-24T21:18:50.837 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134588.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134588.log.gz 2023-12-24T21:18:50.837 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150582.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150582.log.gz 2023-12-24T21:18:50.837 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142817.log 2023-12-24T21:18:50.838 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143228.log 2023-12-24T21:18:50.838 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142817.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142817.log.gz 2023-12-24T21:18:50.838 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143228.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143228.log.gz 2023-12-24T21:18:50.838 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138398.log 2023-12-24T21:18:50.838 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136916.log 2023-12-24T21:18:50.838 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138398.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138398.log.gz 2023-12-24T21:18:50.838 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137835.log 2023-12-24T21:18:50.838 INFO:teuthology.orchestra.run.smithi087.stderr: 54.7% -- replaced with /var/log/ceph/ceph-client.admin.136916.log.gz 2023-12-24T21:18:50.838 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134674.log 2023-12-24T21:18:50.838 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137835.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137835.log.gz 2023-12-24T21:18:50.839 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134674.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134674.log.gz 2023-12-24T21:18:50.839 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139350.log 2023-12-24T21:18:50.839 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131918.log 2023-12-24T21:18:50.839 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139350.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139350.log.gz 2023-12-24T21:18:50.839 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131918.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131918.log.gz 2023-12-24T21:18:50.839 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120090.log 2023-12-24T21:18:50.839 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167524.log 2023-12-24T21:18:50.839 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.120090.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120090.log.gz 2023-12-24T21:18:50.839 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.167524.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167524.log.gz 2023-12-24T21:18:50.839 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131833.log 2023-12-24T21:18:50.839 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131833.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131833.log.gz 2023-12-24T21:18:50.840 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146951.log 2023-12-24T21:18:50.840 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121069.log 2023-12-24T21:18:50.840 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146951.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146951.log.gz 2023-12-24T21:18:50.840 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121069.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121069.log.gz 2023-12-24T21:18:50.840 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137217.log 2023-12-24T21:18:50.840 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131561.log 2023-12-24T21:18:50.840 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137217.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137217.log.gz 2023-12-24T21:18:50.840 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131561.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131561.log.gz 2023-12-24T21:18:50.840 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158707.log 2023-12-24T21:18:50.840 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137424.log 2023-12-24T21:18:50.841 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158707.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158707.log.gz 2023-12-24T21:18:50.841 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137424.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137424.log.gz 2023-12-24T21:18:50.841 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169324.log 2023-12-24T21:18:50.841 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121923.log 2023-12-24T21:18:50.841 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.169324.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169324.log.gz 2023-12-24T21:18:50.841 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121923.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121923.log.gz 2023-12-24T21:18:50.841 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150690.log 2023-12-24T21:18:50.841 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130473.log 2023-12-24T21:18:50.841 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150690.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150690.log.gz 2023-12-24T21:18:50.842 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130473.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130473.log.gz 2023-12-24T21:18:50.842 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121671.log 2023-12-24T21:18:50.842 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160410.log 2023-12-24T21:18:50.842 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121671.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121671.log.gz 2023-12-24T21:18:50.842 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160410.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160410.log.gz 2023-12-24T21:18:50.842 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125314.log 2023-12-24T21:18:50.842 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127010.log 2023-12-24T21:18:50.842 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125314.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125314.log.gz 2023-12-24T21:18:50.842 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127010.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127010.log.gz 2023-12-24T21:18:50.842 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156626.log 2023-12-24T21:18:50.842 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157160.log 2023-12-24T21:18:50.843 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156626.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156626.log.gz 2023-12-24T21:18:50.843 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157160.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157160.log.gz 2023-12-24T21:18:50.843 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121778.log 2023-12-24T21:18:50.843 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159718.log 2023-12-24T21:18:50.843 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121778.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121778.log.gz 2023-12-24T21:18:50.843 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159718.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159718.log.gz 2023-12-24T21:18:50.843 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144435.log 2023-12-24T21:18:50.843 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132270.log 2023-12-24T21:18:50.843 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144435.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144435.log.gz 2023-12-24T21:18:50.843 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132270.log: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.132270.log.gz 2023-12-24T21:18:50.844 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146672.log 2023-12-24T21:18:50.844 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133488.log 2023-12-24T21:18:50.844 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146672.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146672.log.gz 2023-12-24T21:18:50.844 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133488.log: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.133488.log.gz 2023-12-24T21:18:50.844 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161614.log 2023-12-24T21:18:50.844 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126219.log 2023-12-24T21:18:50.844 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161614.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161614.log.gz 2023-12-24T21:18:50.844 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126219.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126219.log.gz 2023-12-24T21:18:50.844 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147382.log 2023-12-24T21:18:50.844 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141600.log 2023-12-24T21:18:50.845 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147382.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147382.log.gz 2023-12-24T21:18:50.845 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141600.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141600.log.gz 2023-12-24T21:18:50.845 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162512.log 2023-12-24T21:18:50.845 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134373.log 2023-12-24T21:18:50.845 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162512.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162512.log.gz 2023-12-24T21:18:50.845 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134373.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134373.log.gz 2023-12-24T21:18:50.845 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120507.log 2023-12-24T21:18:50.845 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166731.log 2023-12-24T21:18:50.845 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.120507.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120507.log.gz 2023-12-24T21:18:50.846 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.166731.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166731.log.gz 2023-12-24T21:18:50.846 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140710.log 2023-12-24T21:18:50.846 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128673.log 2023-12-24T21:18:50.846 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140710.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140710.log.gz 2023-12-24T21:18:50.846 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128673.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128673.log.gz 2023-12-24T21:18:50.846 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138800.log 2023-12-24T21:18:50.846 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132774.log 2023-12-24T21:18:50.846 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138800.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138800.log.gz 2023-12-24T21:18:50.846 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132774.log: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.132774.log.gz 2023-12-24T21:18:50.846 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130073.log 2023-12-24T21:18:50.846 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136074.log 2023-12-24T21:18:50.847 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130073.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130073.log.gz 2023-12-24T21:18:50.847 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136074.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136074.log.gz 2023-12-24T21:18:50.847 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168577.log 2023-12-24T21:18:50.847 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148403.log 2023-12-24T21:18:50.847 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.168577.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168577.log.gz 2023-12-24T21:18:50.847 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148403.log: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.148403.log.gz 2023-12-24T21:18:50.847 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139694.log 2023-12-24T21:18:50.847 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139079.log 2023-12-24T21:18:50.847 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139694.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139694.log.gz 2023-12-24T21:18:50.847 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139079.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139079.log.gz 2023-12-24T21:18:50.848 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136389.log 2023-12-24T21:18:50.848 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136389.log: 55.0% -- replaced with /var/log/ceph/ceph-client.admin.136389.log.gz 2023-12-24T21:18:50.848 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144822.log 2023-12-24T21:18:50.848 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132312.log 2023-12-24T21:18:50.848 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144822.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144822.log.gz 2023-12-24T21:18:50.848 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132312.log: 23.8% -- replaced with /var/log/ceph/ceph-client.admin.132312.log.gz 2023-12-24T21:18:50.848 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139142.log 2023-12-24T21:18:50.848 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164381.log 2023-12-24T21:18:50.848 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139142.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139142.log.gz 2023-12-24T21:18:50.848 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.164381.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164381.log.gz 2023-12-24T21:18:50.849 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125983.log 2023-12-24T21:18:50.849 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144800.log 2023-12-24T21:18:50.849 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125983.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125983.log.gz 2023-12-24T21:18:50.849 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144800.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144800.log.gz 2023-12-24T21:18:50.849 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137277.log 2023-12-24T21:18:50.849 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155415.log 2023-12-24T21:18:50.849 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137277.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137277.log.gz 2023-12-24T21:18:50.849 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155415.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155415.log.gz 2023-12-24T21:18:50.849 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160389.log 2023-12-24T21:18:50.849 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160389.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160389.log.gz 2023-12-24T21:18:50.850 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166292.log 2023-12-24T21:18:50.850 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142985.log 2023-12-24T21:18:50.850 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.166292.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166292.log.gz 2023-12-24T21:18:50.850 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142985.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142985.log.gz 2023-12-24T21:18:50.850 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154723.log 2023-12-24T21:18:50.850 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164321.log 2023-12-24T21:18:50.850 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154723.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154723.log.gz 2023-12-24T21:18:50.850 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.164321.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164321.log.gz 2023-12-24T21:18:50.850 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140774.log 2023-12-24T21:18:50.850 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122388.log 2023-12-24T21:18:50.851 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140774.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140774.log.gz 2023-12-24T21:18:50.851 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122388.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122388.log.gz 2023-12-24T21:18:50.851 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127350.log 2023-12-24T21:18:50.851 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143361.log 2023-12-24T21:18:50.851 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127350.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127350.log.gz 2023-12-24T21:18:50.851 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143361.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143361.log.gz 2023-12-24T21:18:50.851 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170272.log 2023-12-24T21:18:50.851 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153906.log 2023-12-24T21:18:50.851 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.170272.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170272.log.gz 2023-12-24T21:18:50.851 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153906.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153906.log.gz 2023-12-24T21:18:50.852 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131289.log 2023-12-24T21:18:50.852 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172215.log 2023-12-24T21:18:50.852 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131289.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131289.log.gz 2023-12-24T21:18:50.852 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.172215.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.172215.log.gz 2023-12-24T21:18:50.852 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155311.log 2023-12-24T21:18:50.852 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136096.log 2023-12-24T21:18:50.852 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155311.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155311.log.gz 2023-12-24T21:18:50.852 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136096.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136096.log.gz 2023-12-24T21:18:50.852 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147402.log 2023-12-24T21:18:50.853 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163740.log 2023-12-24T21:18:50.853 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147402.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147402.log.gz 2023-12-24T21:18:50.853 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163740.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163740.log.gz 2023-12-24T21:18:50.853 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122254.log 2023-12-24T21:18:50.853 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122254.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.122254.log.gz 2023-12-24T21:18:50.853 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139500.log 2023-12-24T21:18:50.853 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131204.log 2023-12-24T21:18:50.853 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139500.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139500.log.gz 2023-12-24T21:18:50.853 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131204.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131204.log.gz 2023-12-24T21:18:50.853 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158095.log 2023-12-24T21:18:50.854 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121155.log 2023-12-24T21:18:50.854 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158095.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158095.log.gz 2023-12-24T21:18:50.854 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121155.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121155.log.gz 2023-12-24T21:18:50.854 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134076.log 2023-12-24T21:18:50.854 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127265.log 2023-12-24T21:18:50.854 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134076.log: 23.5% -- replaced with /var/log/ceph/ceph-client.admin.134076.log.gz 2023-12-24T21:18:50.854 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127265.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127265.log.gz 2023-12-24T21:18:50.854 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151639.log 2023-12-24T21:18:50.854 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162389.log 2023-12-24T21:18:50.854 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151639.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151639.log.gz 2023-12-24T21:18:50.855 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162389.log: 53.1% -- replaced with /var/log/ceph/ceph-client.admin.162389.log.gz 2023-12-24T21:18:50.855 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148942.log 2023-12-24T21:18:50.855 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135472.log 2023-12-24T21:18:50.855 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148942.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148942.log.gz 2023-12-24T21:18:50.855 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135472.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135472.log.gz 2023-12-24T21:18:50.855 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140983.log 2023-12-24T21:18:50.855 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140983.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140983.log.gz 2023-12-24T21:18:50.855 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137505.log 2023-12-24T21:18:50.855 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138659.log 2023-12-24T21:18:50.855 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137505.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137505.log.gz 2023-12-24T21:18:50.856 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138659.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138659.log.gz 2023-12-24T21:18:50.856 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151467.log 2023-12-24T21:18:50.856 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154866.log 2023-12-24T21:18:50.856 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151467.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151467.log.gz 2023-12-24T21:18:50.856 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154866.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154866.log.gz 2023-12-24T21:18:50.856 INFO:teuthology.orchestra.run.smithi087.stderr: 92.1% -- replaced with /var/log/ceph/ceph.log.gz 2023-12-24T21:18:50.856 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170925.log 2023-12-24T21:18:50.856 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129953.log 2023-12-24T21:18:50.856 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.170925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139823.log 2023-12-24T21:18:50.856 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170925.log.gz 2023-12-24T21:18:50.857 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129953.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129953.log.gz 2023-12-24T21:18:50.857 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140471.log 2023-12-24T21:18:50.857 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139823.log.gz 2023-12-24T21:18:50.857 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121262.log 2023-12-24T21:18:50.857 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159844.log 2023-12-24T21:18:50.857 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140471.log.gz 2023-12-24T21:18:50.857 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144521.log 2023-12-24T21:18:50.857 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121262.log.gz 2023-12-24T21:18:50.857 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134202.log 2023-12-24T21:18:50.857 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159844.log.gz 2023-12-24T21:18:50.858 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144521.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144521.log.gz 2023-12-24T21:18:50.858 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134202.log: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.134202.log.gz 2023-12-24T21:18:50.858 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124519.log 2023-12-24T21:18:50.858 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145711.log 2023-12-24T21:18:50.858 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123512.log 2023-12-24T21:18:50.858 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124519.log.gz 2023-12-24T21:18:50.858 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145711.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145711.log.gz 2023-12-24T21:18:50.858 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123512.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123512.log.gz 2023-12-24T21:18:50.858 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130507.log 2023-12-24T21:18:50.858 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140495.log 2023-12-24T21:18:50.859 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147588.log 2023-12-24T21:18:50.859 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130507.log.gz 2023-12-24T21:18:50.859 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140495.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140495.log.gz 2023-12-24T21:18:50.859 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147588.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147588.log.gz 2023-12-24T21:18:50.859 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127571.log 2023-12-24T21:18:50.859 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139371.log 2023-12-24T21:18:50.859 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138617.log 2023-12-24T21:18:50.859 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127571.log.gz 2023-12-24T21:18:50.859 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139371.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139371.log.gz 2023-12-24T21:18:50.859 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138617.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138617.log.gz 2023-12-24T21:18:50.860 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166936.log 2023-12-24T21:18:50.860 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132879.log 2023-12-24T21:18:50.860 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.166936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172166.log 2023-12-24T21:18:50.860 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166936.log.gz 2023-12-24T21:18:50.860 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132879.log: 23.1% -- replaced with /var/log/ceph/ceph-client.admin.132879.log.gz 2023-12-24T21:18:50.860 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.172166.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.172166.log.gz 2023-12-24T21:18:50.860 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122140.log 2023-12-24T21:18:50.860 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169733.log 2023-12-24T21:18:50.860 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155132.log 2023-12-24T21:18:50.860 INFO:teuthology.orchestra.run.smithi087.stderr: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.122140.log.gz 2023-12-24T21:18:50.861 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.169733.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169733.log.gz 2023-12-24T21:18:50.861 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155132.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155132.log.gz 2023-12-24T21:18:50.861 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131306.log 2023-12-24T21:18:50.861 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136182.log 2023-12-24T21:18:50.861 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125164.log 2023-12-24T21:18:50.861 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131306.log.gz 2023-12-24T21:18:50.861 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136182.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136182.log.gz 2023-12-24T21:18:50.861 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125782.log 2023-12-24T21:18:50.861 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125164.log.gz 2023-12-24T21:18:50.861 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147979.log 2023-12-24T21:18:50.862 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119821.log 2023-12-24T21:18:50.862 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.125782.log.gz 2023-12-24T21:18:50.862 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147979.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147979.log.gz 2023-12-24T21:18:50.862 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.119821.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119821.log.gz 2023-12-24T21:18:50.862 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155893.log 2023-12-24T21:18:50.862 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141922.log 2023-12-24T21:18:50.862 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152444.log 2023-12-24T21:18:50.862 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155893.log.gz 2023-12-24T21:18:50.862 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141037.log 2023-12-24T21:18:50.863 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141922.log.gz 2023-12-24T21:18:50.863 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152444.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152444.log.gz 2023-12-24T21:18:50.863 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155839.log 2023-12-24T21:18:50.863 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133824.log 2023-12-24T21:18:50.863 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141037.log.gz 2023-12-24T21:18:50.863 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155839.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155839.log.gz 2023-12-24T21:18:50.863 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133824.log: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.133824.log.gz 2023-12-24T21:18:50.863 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133845.log 2023-12-24T21:18:50.863 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154551.log 2023-12-24T21:18:50.864 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153885.log 2023-12-24T21:18:50.864 INFO:teuthology.orchestra.run.smithi087.stderr: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.133845.log.gz 2023-12-24T21:18:50.864 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154551.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154551.log.gz 2023-12-24T21:18:50.864 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153885.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153885.log.gz 2023-12-24T21:18:50.864 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157220.log 2023-12-24T21:18:50.864 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153648.log 2023-12-24T21:18:50.864 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157902.log 2023-12-24T21:18:50.864 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157220.log.gz 2023-12-24T21:18:50.864 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153648.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153648.log.gz 2023-12-24T21:18:50.864 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157902.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157902.log.gz 2023-12-24T21:18:50.865 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145389.log 2023-12-24T21:18:50.865 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162371.log 2023-12-24T21:18:50.865 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121090.log 2023-12-24T21:18:50.865 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145389.log.gz 2023-12-24T21:18:50.865 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162371.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162371.log.gz 2023-12-24T21:18:50.865 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121090.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121090.log.gz 2023-12-24T21:18:50.865 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156976.log 2023-12-24T21:18:50.865 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126413.log 2023-12-24T21:18:50.865 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153412.log 2023-12-24T21:18:50.866 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156976.log.gz 2023-12-24T21:18:50.866 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126413.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126413.log.gz 2023-12-24T21:18:50.866 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153412.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153412.log.gz 2023-12-24T21:18:50.866 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133950.log 2023-12-24T21:18:50.866 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147999.log 2023-12-24T21:18:50.866 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133635.log 2023-12-24T21:18:50.866 INFO:teuthology.orchestra.run.smithi087.stderr: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.133950.log.gz 2023-12-24T21:18:50.866 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147999.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147999.log.gz 2023-12-24T21:18:50.866 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133635.log: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.133635.log.gz 2023-12-24T21:18:50.867 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128613.log 2023-12-24T21:18:50.867 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140817.log 2023-12-24T21:18:50.867 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141879.log 2023-12-24T21:18:50.867 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128613.log.gz 2023-12-24T21:18:50.867 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140817.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140817.log.gz 2023-12-24T21:18:50.867 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141879.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141879.log.gz 2023-12-24T21:18:50.867 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143109.log 2023-12-24T21:18:50.867 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144198.log 2023-12-24T21:18:50.867 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156035.log 2023-12-24T21:18:50.868 INFO:teuthology.orchestra.run.smithi087.stderr: 56.6% -- replaced with /var/log/ceph/ceph-client.admin.143109.log.gz 2023-12-24T21:18:50.868 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144198.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144198.log.gz 2023-12-24T21:18:50.868 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156035.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156035.log.gz 2023-12-24T21:18:50.868 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160819.log 2023-12-24T21:18:50.868 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123571.log 2023-12-24T21:18:50.868 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143746.log 2023-12-24T21:18:50.868 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160819.log.gz 2023-12-24T21:18:50.868 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123571.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123571.log.gz 2023-12-24T21:18:50.868 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153326.log 2023-12-24T21:18:50.869 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143746.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143746.log.gz 2023-12-24T21:18:50.869 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136813.log 2023-12-24T21:18:50.869 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138341.log 2023-12-24T21:18:50.869 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153326.log.gz 2023-12-24T21:18:50.869 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147526.log 2023-12-24T21:18:50.869 INFO:teuthology.orchestra.run.smithi087.stderr: 23.9% -- replaced with /var/log/ceph/ceph-client.admin.136813.log.gz 2023-12-24T21:18:50.869 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138341.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138341.log.gz 2023-12-24T21:18:50.869 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147526.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147526.log.gz 2023-12-24T21:18:50.869 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129333.log 2023-12-24T21:18:50.870 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128413.log 2023-12-24T21:18:50.870 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.129333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157980.log 2023-12-24T21:18:50.870 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129333.log.gz 2023-12-24T21:18:50.870 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128413.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128413.log.gz 2023-12-24T21:18:50.870 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157980.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157980.log.gz 2023-12-24T21:18:50.870 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125644.log 2023-12-24T21:18:50.870 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138123.log 2023-12-24T21:18:50.870 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149752.log 2023-12-24T21:18:50.870 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125644.log.gz 2023-12-24T21:18:50.871 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138123.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138123.log.gz 2023-12-24T21:18:50.871 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149752.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149752.log.gz 2023-12-24T21:18:50.871 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125185.log 2023-12-24T21:18:50.871 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138300.log 2023-12-24T21:18:50.871 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164522.log 2023-12-24T21:18:50.871 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125185.log.gz 2023-12-24T21:18:50.871 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138300.log: 31.8% -- replaced with /var/log/ceph/ceph-client.admin.138300.log.gz 2023-12-24T21:18:50.871 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.164522.log: 81.7% -- replaced with /var/log/ceph/ceph-client.admin.164522.log.gz 2023-12-24T21:18:50.871 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141089.log 2023-12-24T21:18:50.872 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125765.log 2023-12-24T21:18:50.872 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138842.log 2023-12-24T21:18:50.872 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141089.log.gz 2023-12-24T21:18:50.872 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125765.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125765.log.gz 2023-12-24T21:18:50.872 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120605.log 2023-12-24T21:18:50.872 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138842.log: 0.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.138842.log.gz--verbose 2023-12-24T21:18:50.872 INFO:teuthology.orchestra.run.smithi087.stderr:-- /var/log/ceph/ceph-client.admin.127996.log 2023-12-24T21:18:50.872 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.120605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124176.log 2023-12-24T21:18:50.872 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120605.log.gz 2023-12-24T21:18:50.873 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127996.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127996.log.gz 2023-12-24T21:18:50.873 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124176.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124176.log.gz 2023-12-24T21:18:50.873 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135838.log 2023-12-24T21:18:50.873 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147485.log 2023-12-24T21:18:50.873 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147464.log 2023-12-24T21:18:50.873 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135838.log.gz 2023-12-24T21:18:50.873 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147485.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147485.log.gz 2023-12-24T21:18:50.873 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147464.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147464.log.gz 2023-12-24T21:18:50.873 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135190.log 2023-12-24T21:18:50.874 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150172.log 2023-12-24T21:18:50.874 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131085.log 2023-12-24T21:18:50.874 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135190.log.gz 2023-12-24T21:18:50.874 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150172.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150172.log.gz 2023-12-24T21:18:50.874 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131085.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131085.log.gz 2023-12-24T21:18:50.874 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141449.log 2023-12-24T21:18:50.874 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161858.log 2023-12-24T21:18:50.874 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128132.log 2023-12-24T21:18:50.874 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141449.log.gz 2023-12-24T21:18:50.875 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161858.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161858.log.gz 2023-12-24T21:18:50.875 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128132.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128132.log.gz 2023-12-24T21:18:50.875 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151618.log 2023-12-24T21:18:50.875 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157452.log 2023-12-24T21:18:50.875 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119970.log 2023-12-24T21:18:50.875 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151618.log.gz 2023-12-24T21:18:50.875 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157452.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157452.log.gz 2023-12-24T21:18:50.875 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.119970.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119970.log.gz 2023-12-24T21:18:50.875 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138779.log 2023-12-24T21:18:50.875 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123896.log 2023-12-24T21:18:50.876 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125609.log 2023-12-24T21:18:50.876 INFO:teuthology.orchestra.run.smithi087.stderr: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.138779.log.gz 2023-12-24T21:18:50.876 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123896.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123896.log.gz 2023-12-24T21:18:50.876 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125609.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125609.log.gz 2023-12-24T21:18:50.876 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123731.log 2023-12-24T21:18:50.876 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123404.log 2023-12-24T21:18:50.876 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157374.log 2023-12-24T21:18:50.876 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123731.log.gz 2023-12-24T21:18:50.876 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123404.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123404.log.gz 2023-12-24T21:18:50.877 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157374.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157374.log.gz 2023-12-24T21:18:50.877 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137055.log 2023-12-24T21:18:50.877 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134997.log 2023-12-24T21:18:50.877 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141707.log 2023-12-24T21:18:50.877 INFO:teuthology.orchestra.run.smithi087.stderr: 9.3% -- replaced with /var/log/ceph/ceph-client.admin.137055.log.gz 2023-12-24T21:18:50.877 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134997.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134997.log.gz 2023-12-24T21:18:50.877 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141707.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141707.log.gz 2023-12-24T21:18:50.877 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148511.log 2023-12-24T21:18:50.877 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146693.log 2023-12-24T21:18:50.877 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151446.log 2023-12-24T21:18:50.878 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148511.log.gz 2023-12-24T21:18:50.878 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156742.log 2023-12-24T21:18:50.878 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146693.log.gz 2023-12-24T21:18:50.878 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151446.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151446.log.gz 2023-12-24T21:18:50.878 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156742.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156742.log.gz 2023-12-24T21:18:50.878 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127282.log 2023-12-24T21:18:50.878 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163006.log 2023-12-24T21:18:50.878 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134034.log 2023-12-24T21:18:50.878 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127282.log.gz 2023-12-24T21:18:50.878 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163006.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163006.log.gz 2023-12-24T21:18:50.879 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134034.log: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.134034.log.gz 2023-12-24T21:18:50.879 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162449.log 2023-12-24T21:18:50.879 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167834.log 2023-12-24T21:18:50.879 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126348.log 2023-12-24T21:18:50.879 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162449.log.gz 2023-12-24T21:18:50.879 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.167834.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167834.log.gz 2023-12-24T21:18:50.879 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126348.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126348.log.gz 2023-12-24T21:18:50.879 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153584.log 2023-12-24T21:18:50.879 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124798.log 2023-12-24T21:18:50.880 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163547.log 2023-12-24T21:18:50.880 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153584.log.gz 2023-12-24T21:18:50.880 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124798.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124798.log.gz 2023-12-24T21:18:50.880 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163547.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163547.log.gz 2023-12-24T21:18:50.880 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124368.log 2023-12-24T21:18:50.880 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135795.log 2023-12-24T21:18:50.880 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154530.log 2023-12-24T21:18:50.880 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124368.log.gz 2023-12-24T21:18:50.880 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135795.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135795.log.gz 2023-12-24T21:18:50.880 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154530.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154530.log.gz 2023-12-24T21:18:50.881 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130694.log 2023-12-24T21:18:50.881 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150516.log 2023-12-24T21:18:50.881 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171970.log 2023-12-24T21:18:50.881 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130694.log.gz 2023-12-24T21:18:50.881 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150516.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150516.log.gz 2023-12-24T21:18:50.881 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.171970.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171970.log.gz 2023-12-24T21:18:50.881 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164213.log 2023-12-24T21:18:50.881 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134567.log 2023-12-24T21:18:50.881 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.164213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152583.log 2023-12-24T21:18:50.882 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164213.log.gz 2023-12-24T21:18:50.882 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134567.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134567.log.gz 2023-12-24T21:18:50.882 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152583.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152583.log.gz 2023-12-24T21:18:50.882 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122444.log 2023-12-24T21:18:50.882 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166243.log 2023-12-24T21:18:50.882 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134459.log 2023-12-24T21:18:50.882 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122444.log.gz 2023-12-24T21:18:50.882 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.166243.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166243.log.gz 2023-12-24T21:18:50.882 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134459.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134459.log.gz 2023-12-24T21:18:50.882 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147058.log 2023-12-24T21:18:50.883 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158393.log 2023-12-24T21:18:50.883 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152999.log 2023-12-24T21:18:50.883 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147058.log.gz 2023-12-24T21:18:50.883 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158393.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158393.log.gz 2023-12-24T21:18:50.883 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152999.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152999.log.gz 2023-12-24T21:18:50.883 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150561.log 2023-12-24T21:18:50.883 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127435.log 2023-12-24T21:18:50.883 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161098.log 2023-12-24T21:18:50.883 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150561.log.gz 2023-12-24T21:18:50.883 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162331.log 2023-12-24T21:18:50.884 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127435.log.gz 2023-12-24T21:18:50.884 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161098.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161098.log.gz 2023-12-24T21:18:50.884 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162331.log: 12.0% -- replaced with /var/log/ceph/ceph-client.admin.162331.log.gz 2023-12-24T21:18:50.884 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147712.log 2023-12-24T21:18:50.884 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154229.log 2023-12-24T21:18:50.884 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169275.log 2023-12-24T21:18:50.884 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147712.log.gz 2023-12-24T21:18:50.884 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154229.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154229.log.gz 2023-12-24T21:18:50.884 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.169275.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169275.log.gz 2023-12-24T21:18:50.885 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145518.log 2023-12-24T21:18:50.885 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151221.log 2023-12-24T21:18:50.885 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143767.log 2023-12-24T21:18:50.885 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145518.log.gz 2023-12-24T21:18:50.885 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151221.log: 23.4% -- replaced with /var/log/ceph/ceph-client.admin.151221.log.gz 2023-12-24T21:18:50.885 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143767.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143767.log.gz 2023-12-24T21:18:50.885 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122596.log 2023-12-24T21:18:50.885 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143939.log 2023-12-24T21:18:50.885 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123982.log 2023-12-24T21:18:50.885 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122596.log.gz 2023-12-24T21:18:50.886 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143939.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143939.log.gz 2023-12-24T21:18:50.886 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123982.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123982.log.gz 2023-12-24T21:18:50.886 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158893.log 2023-12-24T21:18:50.886 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149732.log 2023-12-24T21:18:50.886 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154014.log 2023-12-24T21:18:50.886 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158893.log.gz 2023-12-24T21:18:50.886 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127180.log 2023-12-24T21:18:50.886 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149732.log.gz 2023-12-24T21:18:50.886 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154014.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154014.log.gz 2023-12-24T21:18:50.887 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127180.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127180.log.gz 2023-12-24T21:18:50.887 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137297.log 2023-12-24T21:18:50.887 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142347.log 2023-12-24T21:18:50.887 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164401.log 2023-12-24T21:18:50.887 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137297.log.gz 2023-12-24T21:18:50.887 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146543.log 2023-12-24T21:18:50.887 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142347.log.gz 2023-12-24T21:18:50.887 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.164401.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164401.log.gz 2023-12-24T21:18:50.887 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146543.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146543.log.gz 2023-12-24T21:18:50.887 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149329.log 2023-12-24T21:18:50.888 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132354.log 2023-12-24T21:18:50.888 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134610.log 2023-12-24T21:18:50.888 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149329.log.gz 2023-12-24T21:18:50.888 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132354.log: 23.1% -- replaced with /var/log/ceph/ceph-client.admin.132354.log.gz 2023-12-24T21:18:50.888 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134610.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134610.log.gz 2023-12-24T21:18:50.888 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140581.log 2023-12-24T21:18:50.888 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156053.log 2023-12-24T21:18:50.888 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135231.log 2023-12-24T21:18:50.888 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140581.log.gz 2023-12-24T21:18:50.888 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156053.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156053.log.gz 2023-12-24T21:18:50.889 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135231.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135231.log.gz 2023-12-24T21:18:50.889 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162748.log 2023-12-24T21:18:50.889 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136369.log 2023-12-24T21:18:50.889 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162748.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162748.log.gz 2023-12-24T21:18:50.889 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136369.log: 17.2% -- replaced with /var/log/ceph/ceph-client.admin.136369.log.gz 2023-12-24T21:18:50.889 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127537.log 2023-12-24T21:18:50.889 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155548.log 2023-12-24T21:18:50.889 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148985.log 2023-12-24T21:18:50.889 INFO:teuthology.orchestra.run.smithi087.stderr: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127537.log.gz 2023-12-24T21:18:50.890 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155548.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155548.log.gz 2023-12-24T21:18:50.890 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148985.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148985.log.gz 2023-12-24T21:18:50.890 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128893.log 2023-12-24T21:18:50.890 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149587.log 2023-12-24T21:18:50.890 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122482.log 2023-12-24T21:18:50.890 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128893.log.gz 2023-12-24T21:18:50.890 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149587.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149587.log.gz 2023-12-24T21:18:50.890 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132207.log 2023-12-24T21:18:50.890 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122482.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122482.log.gz 2023-12-24T21:18:50.890 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167573.log 2023-12-24T21:18:50.891 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171774.log 2023-12-24T21:18:50.891 INFO:teuthology.orchestra.run.smithi087.stderr: 23.5% -- replaced with /var/log/ceph/ceph-client.admin.132207.log.gz 2023-12-24T21:18:50.891 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.167573.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167573.log.gz 2023-12-24T21:18:50.891 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.171774.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171774.log.gz 2023-12-24T21:18:50.891 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125744.log 2023-12-24T21:18:50.891 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124626.log 2023-12-24T21:18:50.891 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157827.log 2023-12-24T21:18:50.891 INFO:teuthology.orchestra.run.smithi087.stderr: 22.8% -- replaced with /var/log/ceph/ceph-client.admin.125744.log.gz 2023-12-24T21:18:50.891 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136792.log 2023-12-24T21:18:50.892 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124626.log.gz 2023-12-24T21:18:50.892 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157827.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157827.log.gz 2023-12-24T21:18:50.892 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136792.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136792.log.gz 2023-12-24T21:18:50.892 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149981.log 2023-12-24T21:18:50.892 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161270.log 2023-12-24T21:18:50.892 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134416.log 2023-12-24T21:18:50.892 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149981.log.gz 2023-12-24T21:18:50.892 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161270.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161270.log.gz 2023-12-24T21:18:50.892 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134416.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134416.log.gz 2023-12-24T21:18:50.892 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144327.log 2023-12-24T21:18:50.893 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152210.log 2023-12-24T21:18:50.893 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162351.log 2023-12-24T21:18:50.893 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144327.log.gz 2023-12-24T21:18:50.893 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152210.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152210.log.gz 2023-12-24T21:18:50.893 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162351.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162351.log.gz 2023-12-24T21:18:50.893 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124583.log 2023-12-24T21:18:50.893 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126499.log 2023-12-24T21:18:50.893 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159986.log 2023-12-24T21:18:50.893 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124583.log.gz 2023-12-24T21:18:50.893 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126499.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126499.log.gz 2023-12-24T21:18:50.894 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159986.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159986.log.gz 2023-12-24T21:18:50.894 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163504.log 2023-12-24T21:18:50.894 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139000.log 2023-12-24T21:18:50.894 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158292.log 2023-12-24T21:18:50.894 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163504.log.gz 2023-12-24T21:18:50.894 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139000.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139000.log.gz 2023-12-24T21:18:50.894 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158292.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158292.log.gz 2023-12-24T21:18:50.894 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135515.log 2023-12-24T21:18:50.894 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142159.log 2023-12-24T21:18:50.895 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151262.log 2023-12-24T21:18:50.895 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135515.log.gz 2023-12-24T21:18:50.895 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142159.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142159.log.gz 2023-12-24T21:18:50.895 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151262.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151262.log.gz 2023-12-24T21:18:50.895 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149135.log 2023-12-24T21:18:50.895 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163805.log 2023-12-24T21:18:50.895 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162173.log 2023-12-24T21:18:50.895 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149135.log.gz 2023-12-24T21:18:50.895 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163805.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163805.log.gz 2023-12-24T21:18:50.895 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134287.log 2023-12-24T21:18:50.896 INFO:teuthology.orchestra.run.smithi087.stderr: 23.5% -- replaced with /var/log/ceph/ceph-client.admin.162173.log.gz 2023-12-24T21:18:50.896 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126520.log 2023-12-24T21:18:50.896 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163076.log 2023-12-24T21:18:50.896 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134287.log.gz 2023-12-24T21:18:50.896 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126520.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126520.log.gz 2023-12-24T21:18:50.896 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163076.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163076.log.gz 2023-12-24T21:18:50.896 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144068.log 2023-12-24T21:18:50.896 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152738.log 2023-12-24T21:18:50.896 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144068.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144068.log.gz 2023-12-24T21:18:50.896 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152738.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152738.log.gz 2023-12-24T21:18:50.897 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136534.log 2023-12-24T21:18:50.897 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149608.log 2023-12-24T21:18:50.897 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128553.log 2023-12-24T21:18:50.897 INFO:teuthology.orchestra.run.smithi087.stderr: 51.1% -- replaced with /var/log/ceph/ceph-client.admin.136534.log.gz 2023-12-24T21:18:50.897 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149608.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149608.log.gz 2023-12-24T21:18:50.897 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128553.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128553.log.gz 2023-12-24T21:18:50.897 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171872.log 2023-12-24T21:18:50.897 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121981.log 2023-12-24T21:18:50.897 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.171872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155491.log 2023-12-24T21:18:50.898 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171872.log.gz 2023-12-24T21:18:50.898 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121981.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121981.log.gz 2023-12-24T21:18:50.898 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155491.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155491.log.gz 2023-12-24T21:18:50.898 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153261.log 2023-12-24T21:18:50.898 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158059.log 2023-12-24T21:18:50.898 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155762.log 2023-12-24T21:18:50.898 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153261.log.gz 2023-12-24T21:18:50.898 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158059.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158059.log.gz 2023-12-24T21:18:50.898 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155762.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155762.log.gz 2023-12-24T21:18:50.898 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141194.log 2023-12-24T21:18:50.899 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160432.log 2023-12-24T21:18:50.899 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158829.log 2023-12-24T21:18:50.899 INFO:teuthology.orchestra.run.smithi087.stderr: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.141194.log.gz 2023-12-24T21:18:50.899 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160432.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160432.log.gz 2023-12-24T21:18:50.899 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158829.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158829.log.gz 2023-12-24T21:18:50.899 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132003.log 2023-12-24T21:18:50.899 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127078.log 2023-12-24T21:18:50.899 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147258.log 2023-12-24T21:18:50.899 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132003.log.gz 2023-12-24T21:18:50.900 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127078.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127078.log.gz 2023-12-24T21:18:50.900 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147258.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147258.log.gz 2023-12-24T21:18:50.900 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124197.log 2023-12-24T21:18:50.900 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140796.log 2023-12-24T21:18:50.900 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149668.log 2023-12-24T21:18:50.900 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124197.log.gz 2023-12-24T21:18:50.900 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147939.log 2023-12-24T21:18:50.900 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140796.log.gz 2023-12-24T21:18:50.900 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149668.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149668.log.gz 2023-12-24T21:18:50.900 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147939.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147939.log.gz 2023-12-24T21:18:50.901 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137982.log 2023-12-24T21:18:50.901 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150754.log 2023-12-24T21:18:50.901 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159001.log 2023-12-24T21:18:50.901 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137982.log.gz 2023-12-24T21:18:50.901 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150754.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150754.log.gz 2023-12-24T21:18:50.901 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159001.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159001.log.gz 2023-12-24T21:18:50.901 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-12-24T21:18:50.901 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130153.log 2023-12-24T21:18:50.901 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122658.log 2023-12-24T21:18:50.902 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130153.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130153.log.gz 2023-12-24T21:18:50.902 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122658.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122658.log.gz 2023-12-24T21:18:50.902 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151897.log 2023-12-24T21:18:50.902 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155077.log 2023-12-24T21:18:50.902 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.151897.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151897.log.gz 2023-12-24T21:18:50.902 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155077.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155077.log.gz 2023-12-24T21:18:50.902 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155565.log 2023-12-24T21:18:50.902 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155332.log 2023-12-24T21:18:50.902 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155565.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155565.log.gz 2023-12-24T21:18:50.903 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155332.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155332.log.gz 2023-12-24T21:18:50.903 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127129.log 2023-12-24T21:18:50.903 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159596.log 2023-12-24T21:18:50.903 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127129.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127129.log.gz 2023-12-24T21:18:50.903 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159596.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159596.log.gz 2023-12-24T21:18:50.903 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163977.log 2023-12-24T21:18:50.903 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124949.log 2023-12-24T21:18:50.903 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163977.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163977.log.gz 2023-12-24T21:18:50.903 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124949.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124949.log.gz 2023-12-24T21:18:50.903 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142180.log 2023-12-24T21:18:50.904 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161184.log 2023-12-24T21:18:50.904 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159884.log 2023-12-24T21:18:50.904 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142180.log.gz 2023-12-24T21:18:50.904 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161184.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161184.log.gz 2023-12-24T21:18:50.904 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159884.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159884.log.gz 2023-12-24T21:18:50.904 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121327.log 2023-12-24T21:18:50.904 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139651.log 2023-12-24T21:18:50.904 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121327.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121327.log.gz 2023-12-24T21:18:50.904 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139651.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139651.log.gz 2023-12-24T21:18:50.904 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145303.log 2023-12-24T21:18:50.905 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142390.log 2023-12-24T21:18:50.905 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145303.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145303.log.gz 2023-12-24T21:18:50.905 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142390.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142390.log.gz 2023-12-24T21:18:50.905 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168734.log 2023-12-24T21:18:50.905 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145029.log 2023-12-24T21:18:50.905 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.168734.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168734.log.gz 2023-12-24T21:18:50.905 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145029.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145029.log.gz 2023-12-24T21:18:50.905 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131493.log 2023-12-24T21:18:50.905 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162576.log 2023-12-24T21:18:50.906 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131493.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131493.log.gz 2023-12-24T21:18:50.906 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162576.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162576.log.gz 2023-12-24T21:18:50.906 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145324.log 2023-12-24T21:18:50.906 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147756.log 2023-12-24T21:18:50.906 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145324.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145324.log.gz 2023-12-24T21:18:50.906 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152895.log 2023-12-24T21:18:50.906 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147756.log.gz 2023-12-24T21:18:50.906 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161881.log 2023-12-24T21:18:50.906 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152895.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152895.log.gz 2023-12-24T21:18:50.907 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161881.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161881.log.gz 2023-12-24T21:18:50.907 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163375.log 2023-12-24T21:18:50.907 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158272.log 2023-12-24T21:18:50.907 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163375.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163375.log.gz 2023-12-24T21:18:50.907 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158272.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158272.log.gz 2023-12-24T21:18:50.907 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170876.log 2023-12-24T21:18:50.907 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132585.log 2023-12-24T21:18:50.907 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.170876.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170876.log.gz 2023-12-24T21:18:50.907 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158413.log 2023-12-24T21:18:50.907 INFO:teuthology.orchestra.run.smithi087.stderr: 23.1% -- replaced with /var/log/ceph/ceph-client.admin.132585.log.gz 2023-12-24T21:18:50.908 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130293.log 2023-12-24T21:18:50.908 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158413.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158413.log.gz 2023-12-24T21:18:50.908 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130293.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130293.log.gz 2023-12-24T21:18:50.908 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140861.log 2023-12-24T21:18:50.908 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147838.log 2023-12-24T21:18:50.908 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.140861.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140861.log.gz 2023-12-24T21:18:50.908 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147838.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147838.log.gz 2023-12-24T21:18:50.908 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137338.log 2023-12-24T21:18:50.908 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156389.log 2023-12-24T21:18:50.909 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137338.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137338.log.gz 2023-12-24T21:18:50.909 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156389.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156389.log.gz 2023-12-24T21:18:50.909 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137877.log 2023-12-24T21:18:50.909 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135945.log 2023-12-24T21:18:50.909 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.137877.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137877.log.gz 2023-12-24T21:18:50.909 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135945.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135945.log.gz 2023-12-24T21:18:50.909 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130173.log 2023-12-24T21:18:50.909 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135730.log 2023-12-24T21:18:50.909 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130173.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130173.log.gz 2023-12-24T21:18:50.910 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135730.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135730.log.gz 2023-12-24T21:18:50.910 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170680.log 2023-12-24T21:18:50.910 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159804.log 2023-12-24T21:18:50.910 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.170680.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170680.log.gz 2023-12-24T21:18:50.910 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159804.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159804.log.gz 2023-12-24T21:18:50.910 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135988.log 2023-12-24T21:18:50.910 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147692.log 2023-12-24T21:18:50.910 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135988.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135988.log.gz 2023-12-24T21:18:50.910 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.147692.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147692.log.gz 2023-12-24T21:18:50.911 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152855.log 2023-12-24T21:18:50.911 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157080.log 2023-12-24T21:18:50.911 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152855.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152855.log.gz 2023-12-24T21:18:50.911 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157080.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157080.log.gz 2023-12-24T21:18:50.911 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156491.log 2023-12-24T21:18:50.911 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144564.log 2023-12-24T21:18:50.911 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156491.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156491.log.gz 2023-12-24T21:18:50.911 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144564.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144564.log.gz 2023-12-24T21:18:50.911 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155665.log 2023-12-24T21:18:50.911 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128653.log 2023-12-24T21:18:50.912 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155665.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155665.log.gz 2023-12-24T21:18:50.912 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128653.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128653.log.gz 2023-12-24T21:18:50.912 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130273.log 2023-12-24T21:18:50.912 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133866.log 2023-12-24T21:18:50.912 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.130273.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130273.log.gz 2023-12-24T21:18:50.912 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133866.log: 23.1% -- replaced with /var/log/ceph/ceph-client.admin.133866.log.gz 2023-12-24T21:18:50.912 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143489.log 2023-12-24T21:18:50.912 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150024.log 2023-12-24T21:18:50.912 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.143489.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143489.log.gz 2023-12-24T21:18:50.912 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150024.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150024.log.gz 2023-12-24T21:18:50.913 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148382.log 2023-12-24T21:18:50.913 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159366.log 2023-12-24T21:18:50.913 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144047.log 2023-12-24T21:18:50.913 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148382.log.gz 2023-12-24T21:18:50.913 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159366.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159366.log.gz 2023-12-24T21:18:50.913 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144047.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144047.log.gz 2023-12-24T21:18:50.913 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131952.log 2023-12-24T21:18:50.913 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158607.log 2023-12-24T21:18:50.913 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131952.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131952.log.gz 2023-12-24T21:18:50.914 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158607.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158607.log.gz 2023-12-24T21:18:50.914 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135369.log 2023-12-24T21:18:50.914 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131238.log 2023-12-24T21:18:50.914 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135369.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135369.log.gz 2023-12-24T21:18:50.914 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131238.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131238.log.gz 2023-12-24T21:18:50.914 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139608.log 2023-12-24T21:18:50.914 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146607.log 2023-12-24T21:18:50.914 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139608.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139608.log.gz 2023-12-24T21:18:50.914 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146607.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146607.log.gz 2023-12-24T21:18:50.915 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125897.log 2023-12-24T21:18:50.915 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142611.log 2023-12-24T21:18:50.915 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125897.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125897.log.gz 2023-12-24T21:18:50.915 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142611.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142611.log.gz 2023-12-24T21:18:50.915 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164484.log 2023-12-24T21:18:50.915 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156702.log 2023-12-24T21:18:50.915 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.164484.log: 57.2% -- replaced with /var/log/ceph/ceph-client.admin.164484.log.gz 2023-12-24T21:18:50.915 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156702.log: 52.9% -- replaced with /var/log/ceph/ceph-client.admin.156702.log.gz 2023-12-24T21:18:50.915 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160068.log 2023-12-24T21:18:50.915 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154250.log 2023-12-24T21:18:50.916 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.160068.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160068.log.gz 2023-12-24T21:18:50.916 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.154250.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154250.log.gz 2023-12-24T21:18:50.916 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123469.log 2023-12-24T21:18:50.916 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132438.log 2023-12-24T21:18:50.916 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123469.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123469.log.gz 2023-12-24T21:18:50.916 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.132438.log: 22.7% -- replaced with /var/log/ceph/ceph-client.admin.132438.log.gz 2023-12-24T21:18:50.916 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145948.log 2023-12-24T21:18:50.916 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145690.log 2023-12-24T21:18:50.916 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145948.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145948.log.gz 2023-12-24T21:18:50.916 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.145690.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145690.log.gz 2023-12-24T21:18:50.917 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155056.log 2023-12-24T21:18:50.917 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155512.log 2023-12-24T21:18:50.917 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155056.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155056.log.gz 2023-12-24T21:18:50.917 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155512.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155512.log.gz 2023-12-24T21:18:50.917 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136431.log 2023-12-24T21:18:50.917 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121391.log 2023-12-24T21:18:50.917 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.136431.log: 57.0% -- replaced with /var/log/ceph/ceph-client.admin.136431.log.gz 2023-12-24T21:18:50.917 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.121391.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121391.log.gz 2023-12-24T21:18:50.917 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161925.log 2023-12-24T21:18:50.918 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120978.log 2023-12-24T21:18:50.918 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161925.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161925.log.gz 2023-12-24T21:18:50.918 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.120978.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120978.log.gz 2023-12-24T21:18:50.918 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134055.log 2023-12-24T21:18:50.918 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157179.log 2023-12-24T21:18:50.918 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134055.log: 23.4% -- replaced with /var/log/ceph/ceph-client.admin.134055.log.gz 2023-12-24T21:18:50.918 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157179.log: 54.1% -- replaced with /var/log/ceph/ceph-client.admin.157179.log.gz 2023-12-24T21:18:50.918 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144864.log 2023-12-24T21:18:50.918 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127027.log 2023-12-24T21:18:50.919 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157686.log 2023-12-24T21:18:50.919 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144864.log.gz 2023-12-24T21:18:50.919 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127027.log: 1.2% -- replaced with /var/log/ceph/ceph-client.admin.127027.log.gz 2023-12-24T21:18:50.919 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.157686.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157686.log.gz 2023-12-24T21:18:50.919 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119675.log 2023-12-24T21:18:50.919 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156996.log 2023-12-24T21:18:50.919 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.119675.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119675.log.gz 2023-12-24T21:18:50.919 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156996.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156996.log.gz 2023-12-24T21:18:50.919 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148167.log 2023-12-24T21:18:50.919 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114352.log 2023-12-24T21:18:50.920 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.148167.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148167.log.gz 2023-12-24T21:18:50.920 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.114352.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114352.log.gz 2023-12-24T21:18:50.920 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127809.log 2023-12-24T21:18:50.920 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126004.log 2023-12-24T21:18:50.920 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.127809.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127809.log.gz 2023-12-24T21:18:50.920 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.126004.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126004.log.gz 2023-12-24T21:18:50.920 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167377.log 2023-12-24T21:18:50.920 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128251.log 2023-12-24T21:18:50.920 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.167377.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167377.log.gz 2023-12-24T21:18:50.920 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128251.log: 2.5% -- replaced with /var/log/ceph/ceph-client.admin.128251.log.gz 2023-12-24T21:18:50.921 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161593.log 2023-12-24T21:18:50.921 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133530.log 2023-12-24T21:18:50.921 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.161593.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161593.log.gz 2023-12-24T21:18:50.921 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133530.log: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.133530.log.gz 2023-12-24T21:18:50.921 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131034.log 2023-12-24T21:18:50.921 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141273.log 2023-12-24T21:18:50.921 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131034.log.gz 2023-12-24T21:18:50.921 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141273.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141273.log.gz 2023-12-24T21:18:50.921 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124992.log 2023-12-24T21:18:50.921 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158958.log 2023-12-24T21:18:50.922 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124992.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124992.log.gz 2023-12-24T21:18:50.922 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.158958.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158958.log.gz 2023-12-24T21:18:50.922 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159616.log 2023-12-24T21:18:50.922 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138201.log 2023-12-24T21:18:50.922 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159616.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159616.log.gz 2023-12-24T21:18:50.922 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138201.log: 56.1% -- replaced with /var/log/ceph/ceph-client.admin.138201.log.gz 2023-12-24T21:18:50.922 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135307.log 2023-12-24T21:18:50.922 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123608.log 2023-12-24T21:18:50.922 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135307.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135307.log.gz 2023-12-24T21:18:50.923 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123608.log: 5.3% -- replaced with /var/log/ceph/ceph-client.admin.123608.log.gz 2023-12-24T21:18:50.923 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125099.log 2023-12-24T21:18:50.923 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131255.log 2023-12-24T21:18:50.923 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.125099.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125099.log.gz 2023-12-24T21:18:50.923 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131255.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131255.log.gz 2023-12-24T21:18:50.923 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163848.log 2023-12-24T21:18:50.923 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153099.log 2023-12-24T21:18:50.923 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163848.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163848.log.gz 2023-12-24T21:18:50.923 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153099.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153099.log.gz 2023-12-24T21:18:50.923 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119943.log 2023-12-24T21:18:50.924 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152796.log 2023-12-24T21:18:50.924 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.119943.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119943.log.gz 2023-12-24T21:18:50.924 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152796.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152796.log.gz 2023-12-24T21:18:50.924 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159925.log 2023-12-24T21:18:50.924 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150301.log 2023-12-24T21:18:50.924 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159925.log: 23.2% -- replaced with /var/log/ceph/ceph-client.admin.159925.log.gz 2023-12-24T21:18:50.924 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150301.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150301.log.gz 2023-12-24T21:18:50.924 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155096.log 2023-12-24T21:18:50.924 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142410.log 2023-12-24T21:18:50.924 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.155096.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155096.log.gz 2023-12-24T21:18:50.925 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.142410.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142410.log.gz 2023-12-24T21:18:50.925 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149028.log 2023-12-24T21:18:50.925 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124669.log 2023-12-24T21:18:50.925 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162251.log 2023-12-24T21:18:50.925 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149028.log.gz 2023-12-24T21:18:50.925 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.124669.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124669.log.gz 2023-12-24T21:18:50.925 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162251.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162251.log.gz 2023-12-24T21:18:50.925 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170827.log 2023-12-24T21:18:50.925 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152307.log 2023-12-24T21:18:50.926 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.170827.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170827.log.gz 2023-12-24T21:18:50.926 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152307.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152307.log.gz 2023-12-24T21:18:50.926 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163186.log 2023-12-24T21:18:50.926 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146908.log 2023-12-24T21:18:50.926 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.163186.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163186.log.gz 2023-12-24T21:18:50.926 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146908.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146908.log.gz 2023-12-24T21:18:50.926 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162194.log 2023-12-24T21:18:50.926 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122974.log 2023-12-24T21:18:50.926 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.162194.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162194.log.gz 2023-12-24T21:18:50.926 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.122974.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122974.log.gz 2023-12-24T21:18:50.927 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156450.log 2023-12-24T21:18:50.927 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141901.log 2023-12-24T21:18:50.927 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156450.log: 54.6% -- replaced with /var/log/ceph/ceph-client.admin.156450.log.gz 2023-12-24T21:18:50.927 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.141901.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141901.log.gz 2023-12-24T21:18:50.927 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149157.log 2023-12-24T21:18:50.927 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138560.log 2023-12-24T21:18:50.927 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.149157.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149157.log.gz 2023-12-24T21:18:50.927 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138560.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138560.log.gz 2023-12-24T21:18:50.927 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144585.log 2023-12-24T21:18:50.927 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153369.log 2023-12-24T21:18:50.928 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.144585.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144585.log.gz 2023-12-24T21:18:50.928 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.153369.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153369.log.gz 2023-12-24T21:18:50.928 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131510.log 2023-12-24T21:18:50.928 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134653.log 2023-12-24T21:18:50.928 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.131510.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131510.log.gz 2023-12-24T21:18:50.928 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.134653.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134653.log.gz 2023-12-24T21:18:50.928 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135601.log 2023-12-24T21:18:50.928 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128753.log 2023-12-24T21:18:50.928 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.135601.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135601.log.gz 2023-12-24T21:18:50.929 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.128753.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128753.log.gz 2023-12-24T21:18:50.929 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139021.log 2023-12-24T21:18:50.929 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139436.log 2023-12-24T21:18:50.929 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139021.log: 23.0% -- replaced with /var/log/ceph/ceph-client.admin.139021.log.gz 2023-12-24T21:18:50.929 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.139436.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139436.log.gz 2023-12-24T21:18:50.929 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123340.log 2023-12-24T21:18:50.929 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138180.log 2023-12-24T21:18:50.929 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.123340.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123340.log.gz 2023-12-24T21:18:50.929 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.138180.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138180.log.gz 2023-12-24T21:18:50.929 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146521.log 2023-12-24T21:18:50.930 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133509.log 2023-12-24T21:18:50.930 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.146521.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146521.log.gz 2023-12-24T21:18:50.930 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.133509.log: 22.4% -- replaced with /var/log/ceph/ceph-client.admin.133509.log.gz 2023-12-24T21:18:50.930 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156094.log 2023-12-24T21:18:50.930 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159517.log 2023-12-24T21:18:50.930 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.156094.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156094.log.gz 2023-12-24T21:18:50.930 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.159517.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159517.log.gz 2023-12-24T21:18:50.930 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169528.log 2023-12-24T21:18:50.930 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152423.log 2023-12-24T21:18:50.930 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.169528.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169528.log.gz 2023-12-24T21:18:50.931 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.152423.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152423.log.gz 2023-12-24T21:18:50.931 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150150.log 2023-12-24T21:18:50.931 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.150150.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150150.log.gz 2023-12-24T21:18:51.256 INFO:teuthology.orchestra.run.smithi087.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-12-24T21:18:54.331 INFO:teuthology.orchestra.run.smithi087.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-12-24T21:18:54.763 INFO:teuthology.orchestra.run.smithi087.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-12-24T21:19:49.538 INFO:teuthology.orchestra.run.smithi087.stderr: 97.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-12-24T21:19:52.205 INFO:teuthology.orchestra.run.smithi087.stderr: 97.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-12-24T21:19:52.207 INFO:teuthology.orchestra.run.smithi087.stderr: 2023-12-24T21:19:52.207 INFO:teuthology.orchestra.run.smithi087.stderr:real 1m2.377s 2023-12-24T21:19:52.207 INFO:teuthology.orchestra.run.smithi087.stderr:user 2m6.325s 2023-12-24T21:19:52.207 INFO:teuthology.orchestra.run.smithi087.stderr:sys 0m6.260s 2023-12-24T21:19:52.208 INFO:tasks.ceph:Archiving logs... 2023-12-24T21:19:52.208 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-24_15:40:05-rbd-wip-yuri4-testing-2023-12-21-0917-quincy-distro-default-smithi/7500245/remote/smithi087/log 2023-12-24T21:19:52.209 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-24T21:20:04.340 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f/qa/tasks/ceph.py", line 1910, in task osd_scrub_pgs(ctx, config) File "/home/teuthworker/src/github.com_ceph_ceph-c_5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f/qa/tasks/ceph.py", line 1305, in osd_scrub_pgs raise RuntimeError('Exiting scrub checking -- not all pgs scrubbed.') RuntimeError: Exiting scrub checking -- not all pgs scrubbed. 2023-12-24T21:20:04.342 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-24T21:20:04.362 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f/qa/tasks/ceph.py", line 1910, in task osd_scrub_pgs(ctx, config) File "/home/teuthworker/src/github.com_ceph_ceph-c_5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f/qa/tasks/ceph.py", line 1305, in osd_scrub_pgs raise RuntimeError('Exiting scrub checking -- not all pgs scrubbed.') RuntimeError: Exiting scrub checking -- not all pgs scrubbed. 2023-12-24T21:20:04.365 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-24T21:20:04.366 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-24T21:20:04.426 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-24T21:20:04.435 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-24T21:20:04.435 DEBUG:teuthology.orchestra.run.smithi087:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-24T21:20:04.458 INFO:teuthology.orchestra.run.smithi087.stderr:bash: ntpq: command not found 2023-12-24T21:20:04.463 INFO:teuthology.orchestra.run.smithi087.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-24T21:20:04.463 INFO:teuthology.orchestra.run.smithi087.stdout:=============================================================================== 2023-12-24T21:20:04.464 INFO:teuthology.orchestra.run.smithi087.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 47 +734us[ +734us] +/- 87ms 2023-12-24T21:20:04.464 INFO:teuthology.orchestra.run.smithi087.stdout:^* hv02.front.sepia.ceph.com 3 8 377 230 -874us[ -892us] +/- 50ms 2023-12-24T21:20:04.464 INFO:teuthology.orchestra.run.smithi087.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 +574us[ +574us] +/- 73ms 2023-12-24T21:20:04.464 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-12-24T21:20:04.464 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-24T21:20:04.503 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-24T21:20:04.504 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-24T21:20:04.574 DEBUG:teuthology.orchestra.run.smithi087:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-24T21:20:04.674 DEBUG:teuthology.orchestra.run.smithi087:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-24T21:20:04.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-24T21:20:04.707 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-24T21:20:04.718 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-24T21:20:04.776 INFO:teuthology.task.internal:Duration was 12839.694342 seconds 2023-12-24T21:20:04.777 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-24T21:20:04.796 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-24T21:20:04.797 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-24T21:20:04.845 INFO:teuthology.orchestra.run.smithi087.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-24T21:20:04.898 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-24T21:20:04.898 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi087.front.sepia.ceph.com 2023-12-24T21:20:04.898 DEBUG:teuthology.orchestra.run.smithi087:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-24T21:20:04.921 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-24T21:20:04.922 DEBUG:teuthology.orchestra.run.smithi087:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-24T21:20:05.001 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-24T21:20:05.002 DEBUG:teuthology.orchestra.run.smithi087:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-24T21:20:05.110 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-24T21:20:05.121 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-24T21:20:05.121 DEBUG:teuthology.orchestra.run.smithi087:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-24T21:20:05.148 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-24T21:20:05.160 DEBUG:teuthology.orchestra.run.smithi087:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-24T21:20:05.215 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern = core 2023-12-24T21:20:05.250 DEBUG:teuthology.orchestra.run.smithi087:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-24T21:20:05.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-24T21:20:05.307 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-24T21:20:05.318 INFO:teuthology.task.internal:Transferring archived files... 2023-12-24T21:20:05.320 DEBUG:teuthology.misc:Transferring archived files from smithi087:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-24_15:40:05-rbd-wip-yuri4-testing-2023-12-21-0917-quincy-distro-default-smithi/7500245/remote/smithi087 2023-12-24T21:20:05.320 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-24T21:20:05.388 INFO:teuthology.task.internal:Removing archive directory... 2023-12-24T21:20:05.389 DEBUG:teuthology.orchestra.run.smithi087:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-24T21:20:05.438 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-24T21:20:05.449 INFO:teuthology.task.internal:Not uploading archives. 2023-12-24T21:20:05.449 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-24T21:20:05.464 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-24T21:20:05.465 DEBUG:teuthology.orchestra.run.smithi087:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-24T21:20:05.496 INFO:teuthology.orchestra.run.smithi087.stdout: 525703 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Dec 24 21:20 /home/ubuntu/cephtest 2023-12-24T21:20:05.496 INFO:teuthology.orchestra.run.smithi087.stdout: 789561 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Dec 24 17:57 /home/ubuntu/cephtest/mnt.0 2023-12-24T21:20:05.497 INFO:teuthology.orchestra.run.smithi087.stdout: 789562 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Dec 24 17:58 /home/ubuntu/cephtest/mnt.0/client.0 2023-12-24T21:20:05.497 INFO:teuthology.orchestra.run.smithi087.stdout: 789603 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Dec 24 17:58 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-24T21:20:05.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-24T21:20:05.498 INFO:teuthology.orchestra.run.smithi087.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-12-24T21:20:05.498 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f/qa/tasks/ceph.py", line 1910, in task osd_scrub_pgs(ctx, config) File "/home/teuthworker/src/github.com_ceph_ceph-c_5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f/qa/tasks/ceph.py", line 1305, in osd_scrub_pgs raise RuntimeError('Exiting scrub checking -- not all pgs scrubbed.') RuntimeError: Exiting scrub checking -- not all pgs scrubbed. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi087 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-12-24T21:20:05.510 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-24T21:20:05.520 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-24T21:20:05.558 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: RuntimeError: Exiting scrub checking -- not all pgs scrubbed. 2023-12-24T21:20:05.558 INFO:teuthology.nuke:Checking targets against current locks 2023-12-24T21:20:05.586 DEBUG:teuthology.nuke:shortname: smithi087 2023-12-24T21:20:05.586 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-24T21:20:05.608 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi087.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-24_15:40:05-rbd-wip-yuri4-testing-2023-12-21-0917-quincy-distro-default-smithi/7500245', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-24 17:30:00.411876', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC3zm7LNATb0NkKyv/ek111PQVxF5d3AKbhUg0JJ5rq0jQEgeD6U4YLaQhyKi/4V9SdDJvED3VWEN53UE54teL0='} 2023-12-24T21:20:05.662 INFO:teuthology.orchestra.console.smithi087:Power off 2023-12-24T21:20:05.662 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-24T21:20:05.709 DEBUG:teuthology.orchestra.console.smithi087:power off output: Chassis Power Control: Down/Off 2023-12-24T21:20:05.710 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-24T21:20:05.735 DEBUG:teuthology.orchestra.console.smithi087:check power output: Chassis Power is on 2023-12-24T21:20:09.735 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-24T21:20:09.848 DEBUG:teuthology.orchestra.console.smithi087:check power output: Chassis Power is on 2023-12-24T21:20:13.851 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-24T21:20:13.966 DEBUG:teuthology.orchestra.console.smithi087:check power output: Chassis Power is off 2023-12-24T21:20:14.066 INFO:teuthology.orchestra.console.smithi087:Power off completed 2023-12-24T21:20:14.172 INFO:teuthology.run:Summary data: description: rbd/cli_v1/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/format-1 msgr-failures/few objectstore/bluestore-stupid pool/small-cache-pool supported-random-distro$/{centos_8} workloads/rbd_cli_generic} duration: 12839.694342136383 failure_reason: 'Command failed (workunit test rbd/cli_generic.sh) on smithi087 with status 124: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/cli_generic.sh''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=da470ffc492d45f08538d0344fcdd745 status: fail success: false 2023-12-24T21:20:14.173 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-24T21:20:14.257 INFO:teuthology.run:FAIL